FileBeat feeding different Elasticsearch index depending on the file read.
I use Filebeat to read multiple log files and I want to push them to Logstash in a way that each log file goes to a specific index.
My 2 log sources:
- Apache
- A custom application
Filebeat config
In order to achieve that, at Filebeat level I have to put a "tag" for Aapache logs, and another tag for my custom application logs:
filebeat.inputs:
- type: filestream
enabled: true
paths:
- /var/log/apache2/access.log
tags: ["vm1_apache_access"]
- type: filestream
enabled: true
paths:
- /home/logs/only
tags: ["vm1_home_only"]
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
output.logstash:
hosts: ["logstash.rktmb.org:15044"]
This instructs Filebeat to push data to "logstash.rktmb.org:15044" but:
- with the tag "vm1_apache_access" if they are the Apache logs
- with the tag "vm1_home_only" if they are the custom application logs
Logstash config
Now that streams are tagged, we are going to differentiate the destination index based on those tags.
We achieve that at logstash level with
input {
beats {
port => "15044"
}
}
filter {
if "vm1_apache_access" in [tags] {
grok {
match => { "message" => "%{GREEDYDATA:message}" }
}
}
if "vm1_home_only" in [tags] {
grok {
match => { "message" => "%{GREEDYDATA:message}" }
}
}
}
output {
if "vm1_apache_access" in [tags] {
elasticsearch {
hosts => ["10.10.23.62:9200"]
index => "vm1-apache-access"
}
}
if "vm1_home_only" in [tags] {
elasticsearch {
hosts => ["10.10.23.62:9200"]
index => "vm1-home-only"
}
}
}
The important part is the "output" section, it is going to decide where to write the logs.
The "if" blocks in the filter section are completely useless for our use case here, I just put them for decoration.