Skip to main content

filebeat multiple elasticsearch index

FileBeat feeding different Elasticsearch index depending on the file read.

I use Filebeat to read multiple log files and I want to push them to Logstash in a way that each log file goes to a specific index.

My 2 log sources:
  • Apache
  • A custom application

Filebeat config

In order to achieve that, at Filebeat level I have to put a "tag" for Aapache logs, and another tag for my custom application logs:

 

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /var/log/apache2/access.log
  tags: ["vm1_apache_access"]
  
- type: filestream
  enabled: true
  paths:
    - /home/logs/only
  tags: ["vm1_home_only"]
  
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

output.logstash:
  hosts: ["logstash.rktmb.org:15044"]

  

This instructs Filebeat to push data to "logstash.rktmb.org:15044" but:

  • with the tag "vm1_apache_access" if they are the Apache logs
  • with the tag "vm1_home_only" if they are the custom application logs

Logstash config 

Now that streams are tagged, we are going to differentiate the destination index based on those tags.

We achieve that at logstash level with

input {
  beats {
    port => "15044"
  }
}
filter {
  if "vm1_apache_access" in [tags] {
    grok {
      match => { "message" => "%{GREEDYDATA:message}" }
    }
  }
  if "vm1_home_only" in [tags] {
    grok {
      match => { "message" => "%{GREEDYDATA:message}" }
    }
  }
}
output {
  if "vm1_apache_access" in [tags] {
    elasticsearch {
      hosts => ["10.10.23.62:9200"]
      index => "vm1-apache-access"
    }
  }
  if "vm1_home_only" in [tags] {
    elasticsearch {
      hosts => ["10.10.23.62:9200"]
      index => "vm1-home-only"
    }
  }
}

    

The important part is the "output" section, it is going to decide where to write the logs.

The "if" blocks in the filter section are completely useless for our use case here, I just put them for decoration.

Popular posts from this blog

npm run build base-href

Using NPM to specify base-href When building an Angular application, people usually use "ng" and pass arguments to that invocation. Typically, when wanting to hard code "base-href" in "index.html", one will issue: ng build --base-href='https://ngx.rktmb.org/foo' I used to build my angular apps through Bamboo or Jenkins and they have a "npm" plugin. I got the habit to build the application with "npm run build" before deploying it. But the development team once asked me to set the "--base-href='https://ngx.rktmb.org/foo'" parameter. npm run build --base-href='https://ngx.rktmb.org/foo did not set the base href in indext.html After looking for a while, I found https://github.com/angular/angular-cli/issues/13560 where it says: You need to use −− to pass arguments to npm scripts. This did the job! The command to issue is then: npm run build -- --base-href='https://ngx.rktmb.org/foo&

Emacs TypeScript Development

Emacs Configuration for Typescript In order to comfortably develop on Node, React or Angular projects with Emacs, TIDE is a good solution. We have TypeScript code highlight (that is the minimum!) and code completion based on the codebase (not only on locally defined and builtins) In order to achieve that: Install Emacs (24+) Install Node Install Typescript (which will provide "tsserver") Install TIDE and some usefull dependencies Configure Emacs to use all those Node is then installed in " /home/mihamina/Apps/node-v12.18.0-linux-x64/bin ": you should add it to your PATH. Installing Typescript is done with: npm install --save typescript @types/browserify After that, "tsserver" will be in " /home/mihamina/node_modules/.bin " Then comes the installation of TIDE: With the Emacs package manager, M-x package-install , install "tide". Do the same for "web-mode", "flycheck", "company" "js2-mode" and "

Jenkins invalid privatekey

Publish over SSH, Message "invalid privatekey:" With quite recent (June-July 2020) installations of Jenkins and OpenSSH, I have the following error message when using the "Deploy overs SSH" Jenkins plug-in and publishing artifacts to the target overs SSH: jenkins.plugins.publish_over.BapPublisherException: Failed to add SSH key. Message [invalid privatekey: [B@d8d395a] This problem seems to be referenced here: https://issues.jenkins-ci.org/browse/JENKINS-57495 Just regenerate a key with the right parameters To solve it: ssh-keygen -t rsa -b 4096 Or ssh-keygen -t rsa -b 4096 -m PEM