Basic problems with filebeat

[Dimitri]

Hello, everyone,
I’m moving an old Graylog instance from 2.2.3 to Graylog 4.0.5 + Elasticsearch 7.10.2. There’s a considerable number of servers running an old version of ‘collector-sidecar’.

The first issue I faced was that the names of the custom fields configured in Beats inputs get renamed with prefix fields-. So, app=system becomes fields-app=system. The solutions offered here and here seem to apply to new sidecars only.

So, I tried to configure a new sidecar on one of the servers. sidecar.yml includes only the most necessary information: server_url and server_api_token. Then, I created a configuration called ‘system’. I changed paths to logs, set up one field and changed the Graylog server URL:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

filebeat.inputs:
- input_type: log
  paths:
    - /var/log/audit/audit.log
    - /var/log/cron
    - /var/log/messages
    - /var/log/zabbix/zabbix_agentd.log
    - /var/log/yum.log
  type: log
  fields:
    app: system
output.logstash:
   hosts: ["graylog.mydomain.com:5044"]
path:
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

When I tried to run the new sidecar, it was assigned this configuration, but was in the ‘Failing’ state. I fixed one of the errors by replacing -c %s in the log collector configuration with the exact path to /etc/graylog/sidecar/generated/sidecar.yml, but it still fails with the message Exiting: No prospectors defined.

I can go on with fixing the configuration, but somehow it doesn’t feel right. This is not what I read in Step-by-step guide. Should I go on fixing the errors as they appear or have I already lost my way?
Thanks.

[reimlima (Dummy)]

Hi @Dimitri

where does this configuration came from? your sidecar.yml in your server or Graylog UI?

I’m asking because your error message is complaining about an option that is not in your configuration, I’m wondering if those files are the same.

[Dimitri]

This configuration is taken from Graylog UI (System - Sidecars - Configuration - my configuration - Edit).
As for sidecar.yml, it consists of only two lines, as I mentioned: server_url and server_api_token.
Here is the resulting filebeat.yml that causes the error, I assume:

filebeat:
  prospectors: []
output:
  logstash:
    hosts:
    - localhost:5044
path:
  data: /var/cache/graylog/collector-sidecar/filebeat/data
  logs: /var/log/graylog/collector-sidecar
tags:
- linux
- mx-hd-5.7

[reimlima (Dummy)]

As I suspected, those aren’t the same files. To work properly, it must be.

Via your graylog UI you could be able to send the right file to your server, or you can try to edit this file manually to see if it runs with no errors.

To edit your file manually keep in mind you need to replace this fields with the righ values:

fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

Those informations you find in your Graylog UI following the path System > Sidecars

[Dimitri]

I think I know what I did wrong. Your answer helped me understand that:

1. Changes in configuration can be applied
2. collector-sidecar cannot be installed together with sidecar/filebeat, because collector-sidecar includes its own copy of filebeat
3. filebeat configuration is not stored in /etc/graylog/sidecar/generated/filebeat.yml, but in /var/lib/graylog-sidecar/generated/filebeat.conf

Thank you.

However, I still can’t understand the difference between these two configuration files and why /etc/graylog/sidecar/generated/filebeat.yml is not re-created after changes are made in configuration?

[reimlima (Dummy)]

At this point, I think I can’t help you because I’m relative new with graylog to tell how things work under the hood.

Maybe @aaronsachs can.

[tmacgbay]

On my ubuntu linux system, there is no directory /etc/graylog/sidecar/generated. My guess is that is a legacy directory for the older collectors. I have sidecar 1.0.2 running and see the following in those directories:

# ls -l /etc/graylog/sidecar/
total 8
-rw-r--r-- 1 root root   36 Jul 23  2020 node-id
-rw-r--r-- 1 root root 3080 Jul 23  2020 sidecar.yml

This sidecar.yml is the core settings for sidecar configuration (server_url, server_api_token…etc…)

# ls -l /var/lib/graylog-sidecar/generated/
total 4
-rw------- 1 root root 539 Mar 31 09:49 filebeat.conf

This filebeat.conf is the configuration pushed from Graylog that tells the sidecar what and how to send stuff back to Graylog. Here is a simple one I built in our env that is pushed out from Graylog for you to compare:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: LINUX01
fields.gl2_source_collector: e0593f00-6e4a-4b66-8895-343a57767689

filebeat.inputs:
- type: log
  enabled: true
  exclude_lines: ['cron\:session']
  paths:
    - /var/log/auth.log
  ignore_older: 72h
  tags:
    - linux
  fields:
    beats_os: linux

output.logstash:
   hosts:
   - GRAYLOGl:5044

path:
  data: /var/cache/graylog-sidecar/filebeat/data
  logs: /var/log/graylog-sidecar

Here is what it looks like on the Graylog side:

Hope that helps to look at in comparison to your setup. If there are other parts you would like to see I can post.

[Dimitri]

I’ve just installed graylog-sidecar on another server and there are no files in /etc/graylog/sidecar/generated, you are right.
Thank you, folks, for your extensive answers.