Use sidecar tags in filebeat config

[mpibpc-mroose (Marco Roose)]

1. Describe your incident:
I’m trying to work with modular configurations for my filebeat sidecars. For example I want to include an nginx related filestream only, when the sidecar has the “nginx” tag set. For that I have configured the nginx tag in my sidecar.conf:

tags:
  - linux
  - nginx

… added the tag in the Graylog UI and used it as following in the sidecar configuration:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
<#if sidecar.tags.apache??>
- type: filestream
  id: apache_logs
  paths:
    - /var/log/apache2/error.log
    - /var/log/apache2/*access.log
  fields.apache: true
</#if>
<#if sidecar.tags.nginx??>
- type: filestream
  id: nginx_logs
  paths:
    - /var/log/nginx/access.log
    - /var/log/nginx/error.log
  fields.nginx: true
</#if>
- type: filestream
  id: auth_logs
  paths:
    - /var/log/auth.log
  fields.linux_auth: true
- type: filestream
  id: linux_generic_logs
  paths:
    - /var/log/syslog
    - /var/log/*.log
  prospector.scanner.exclude_files: ['auth.log']
  exclude_lines: ['uccee', 'ucces', 'Started Time & Date Service.']

output.logstash:
   hosts: ["S1020-graylog.mpinat.mpg.de:5044"]
path:
  data: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/data
  logs: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/log

But the nginx “section” is not in my sidecars generated filebeat configuration:

~# cat /var/lib/graylog-sidecar/generated/filebeat.conf

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: S1020-Graylog.mpinat.mpg.de
fields.gl2_source_collector: s1020-graylog

filebeat.inputs:
- type: filestream
  id: auth_logs
  paths:
    - /var/log/auth.log
  fields.linux_auth: true
- type: filestream
  id: linux_generic_logs
  paths:
    - /var/log/syslog
    - /var/log/*.log
  prospector.scanner.exclude_files: ['auth.log']
  exclude_lines: ['uccee', 'ucces', 'Started Time & Date Service.']

output.logstash:
   hosts: ["S1020-graylog.mpinat.mpg.de:5044"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

2. Describe your environment:

• OS Information: Ubuntu 20.04 LTS with dockerrized Graylog 5.02

4. How can the community help?

• Does anyone have something like this working?
• does any documentation about the template engine syntax (<#if …> and so on) exists somewhere?

[benoitp]

Hello a tried something similar and this is not working

In “Collector Configuration”, in one conf I added “test” in “Configuration Assignment Tags”
and in “Configuraton” I added this :

<#if sidecar.tags.test??>
# test
</#if>

On the client side :
/etc/graylog/sidecar/sidecar.yml

tags:
  - test

But nothing happened in “/var/lib/graylog-sidecar/generated/filebeat.conf”

The documentation of Sidecar is not clear at all about tags and how config files are applied if we have several tags on an host who match several configurations.

[tmacgbay]

The sidecar is just a wrapper around getting a Elasticsearch filebeat (Or Nxlog ) configuration out to machines in a consistent manner. I have never seen conditionals <#if... used in a beats configuration (Which… actually doesn’t say much…) … I also can’t find its use in the filebeat documentation. Where are you finding examples of using it - because it would be pretty cool if we could get that working!

[mpibpc-mroose (Marco Roose)]

Seems to be deprecated

Allow multiple configurations for a sidecar

Pleas upvote this issue. I think this is an essential feature which should be implemented again.

[benoitp]

All it comes from the doc that is not clear about this.
I copy/paste the doc with “1.2” written in the commands.

You need to have at least 1.3 version of Sidecar to have it works.

The doc about tags and sniplets is too minimalist.

The things to know :
each Configuration must be a full configuration, as Sidecar will spawn one process of the log shipper for each Configuration attached to the host.

Ex. in “sidecar.yml” on my Linux host, i have this

tags:
  - "Linux standard logs"
  - "Graylog server"

I have 2 corresponding Configuration in Graylog (with the same tag as their name in their conf)

Sidecar automatically launch 2 process of Filebeat with theses configurations :