Skip to main content

filebeat multiple elasticsearch index

FileBeat feeding different Elasticsearch index depending on the file read.

I use Filebeat to read multiple log files and I want to push them to Logstash in a way that each log file goes to a specific index.

My 2 log sources:
  • Apache
  • A custom application

Filebeat config

In order to achieve that, at Filebeat level I have to put a "tag" for Aapache logs, and another tag for my custom application logs:

 

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /var/log/apache2/access.log
  tags: ["vm1_apache_access"]
  
- type: filestream
  enabled: true
  paths:
    - /home/logs/only
  tags: ["vm1_home_only"]
  
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

output.logstash:
  hosts: ["logstash.rktmb.org:15044"]

  

This instructs Filebeat to push data to "logstash.rktmb.org:15044" but:

  • with the tag "vm1_apache_access" if they are the Apache logs
  • with the tag "vm1_home_only" if they are the custom application logs

Logstash config 

Now that streams are tagged, we are going to differentiate the destination index based on those tags.

We achieve that at logstash level with

input {
  beats {
    port => "15044"
  }
}
filter {
  if "vm1_apache_access" in [tags] {
    grok {
      match => { "message" => "%{GREEDYDATA:message}" }
    }
  }
  if "vm1_home_only" in [tags] {
    grok {
      match => { "message" => "%{GREEDYDATA:message}" }
    }
  }
}
output {
  if "vm1_apache_access" in [tags] {
    elasticsearch {
      hosts => ["10.10.23.62:9200"]
      index => "vm1-apache-access"
    }
  }
  if "vm1_home_only" in [tags] {
    elasticsearch {
      hosts => ["10.10.23.62:9200"]
      index => "vm1-home-only"
    }
  }
}

    

The important part is the "output" section, it is going to decide where to write the logs.

The "if" blocks in the filter section are completely useless for our use case here, I just put them for decoration.

Popular posts from this blog

npm run build base-href

Using NPM to specify base-href When building an Angular application, people usually use "ng" and pass arguments to that invocation. Typically, when wanting to hard code "base-href" in "index.html", one will issue: ng build --base-href='https://ngx.rktmb.org/foo' I used to build my angular apps through Bamboo or Jenkins and they have a "npm" plugin. I got the habit to build the application with "npm run build" before deploying it. But the development team once asked me to set the "--base-href='https://ngx.rktmb.org/foo'" parameter. npm run build --base-href='https://ngx.rktmb.org/foo did not set the base href in indext.html After looking for a while, I found https://github.com/angular/angular-cli/issues/13560 where it says: You need to use −− to pass arguments to npm scripts. This did the job! The command to issue is then: npm run build -- --base-href='https://ngx.rktmb.org/foo&

Jenkins invalid privatekey

Publish over SSH, Message "invalid privatekey:" With quite recent (June-July 2020) installations of Jenkins and OpenSSH, I have the following error message when using the "Deploy overs SSH" Jenkins plug-in and publishing artifacts to the target overs SSH: jenkins.plugins.publish_over.BapPublisherException: Failed to add SSH key. Message [invalid privatekey: [B@d8d395a] This problem seems to be referenced here: https://issues.jenkins-ci.org/browse/JENKINS-57495 Just regenerate a key with the right parameters To solve it: ssh-keygen -t rsa -b 4096 Or ssh-keygen -t rsa -b 4096 -m PEM

VMWare Keyboard Latency

Workstation VM UI lag when typing When using a VMWare Workstation VM, I noticed there is a latency when typing in the keyboard and the real appearance of the typed character. I searched and found: Noticeable typing lag in Linux VM terminals since v16.2 upgrade on Linux host To make it short, what solved it for me: Disable 3D acceleration in the VM setting .