Skip to main content

filebeat multiple elasticsearch index

FileBeat feeding different Elasticsearch index depending on the file read.

I use Filebeat to read multiple log files and I want to push them to Logstash in a way that each log file goes to a specific index.

My 2 log sources:
  • Apache
  • A custom application

Filebeat config

In order to achieve that, at Filebeat level I have to put a "tag" for Aapache logs, and another tag for my custom application logs:

 

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /var/log/apache2/access.log
  tags: ["vm1_apache_access"]
  
- type: filestream
  enabled: true
  paths:
    - /home/logs/only
  tags: ["vm1_home_only"]
  
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

output.logstash:
  hosts: ["logstash.rktmb.org:15044"]

This instructs Filebeat to push data to "logstash.rktmb.org:15044" but:

  • with the tag "vm1_apache_access" if they are the Apache logs
  • with the tag "vm1_home_only" if they are the custom application logs

Logstash config 

Now that streams are tagged, we are going to differentiate the destination index based on those tags.

We achieve that at logstash level with

input {
  beats {
    port => "15044"
  }
}
filter {
  if "vm1_apache_access" in [tags] {
    grok {
      match => { "message" => "%{GREEDYDATA:message}" }
    }
  }
  if "vm1_home_only" in [tags] {
    grok {
      match => { "message" => "%{GREEDYDATA:message}" }
    }
  }
}
output {
  if "vm1_apache_access" in [tags] {
    elasticsearch {
      hosts => ["10.10.23.62:9200"]
      index => "vm1-apache-access"
    }
  }
  if "vm1_home_only" in [tags] {
    elasticsearch {
      hosts => ["10.10.23.62:9200"]
      index => "vm1-home-only"
    }
  }
}

The important part is the "output" section, it is going to decide where to write the logs.

The "if" blocks in the filter section are completely useless for our use case here, I just put them for decoration.

Popular posts from this blog

npm run build base-href

Using NPM to specify base-href When building an Angular application, people usually use "ng" and pass arguments to that invocation. Typically, when wanting to hard code "base-href" in "index.html", one will issue: ng build --base-href='https://ngx.rktmb.org/foo' I used to build my angular apps through Bamboo or Jenkins and they have a "npm" plugin. I got the habit to build the application with "npm run build" before deploying it. But the development team once asked me to set the "--base-href='https://ngx.rktmb.org/foo'" parameter. npm run build --base-href='https://ngx.rktmb.org/foo did not set the base href in indext.html After looking for a while, I found https://github.com/angular/angular-cli/issues/13560 where it says: You need to use −− to pass arguments to npm scripts. This did the job! The command to issue is then: npm run build -- --base-href='https://ngx.rktmb.org/foo...
Read more

wget maven ntlm proxy

How to make wget, curl and Maven download behind an NTLM Proxy Working on CentOS, behind an NTLM proxy: yum can deal without problem with a NTLM Proxy wget, curl and Maven cannot The solution is to use " cntlm ". " cntlm " is a NTLM client for proxies requiring NTLM authentication. How it works Install "cntlm" Configure "cntlm"  by giving it your credentials by giving it the NTLM Proxy Start "cntlm" deamon (it listens to "127.0.0.1:3128") Configure wget, curl and Maven to use "cntlm" instead of using directly the NTLM Proxy Note: You will have then a kind of 2 stages Proxy : cntlm + the NTLM proxy Configure CNTLM After installing cntlm, the configuration file is in "cntlm.conf". You must have your domain (in the Windows meaning), proxy login and  proxy password. Mine are respectively: rktmb.org, mihamina, 1234abcd (yes, just for the example) You must have you NTLM Proxy Hostnama or IP ...
Read more

Undefined global vim

Defining vim as global outside of Neovim When developing plugins for Neovim, particularly in Lua, developers often encounter the "Undefined global vim" warning. This warning can be a nuisance and disrupt the development workflow. However, there is a straightforward solution to this problem by configuring the Lua Language Server Protocol (LSP) to recognize 'vim' as a global variable. Getting "Undefined global vim" warning when developing Neovim plugin While developing Neovim plugins using Lua, the Lua language server might not recognize the 'vim' namespace by default. This leads to warnings about 'vim' being an undefined global variable. These warnings are not just annoying but can also clutter the development environment with unnecessary alerts, potentially hiding other important warnings or errors. Defining vim as global in Lua LSP configuration to get rid of the warning To resolve the "Undefined global vi...
Read more