Sidecar 1.4.0 with filebeat doesn't change path.data & path.logs

[zyxep]

Hey,

I am using filebeat 8.9 with graylog-sidecar 1.4.0 on ubuntu servers.
when the systemd service graylog-sidecar starts, it starts a process like:

/usr/share/filebeat/bin/filebeat --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat -c /var/lib/graylog-sidecar/generated/648a0c4345c6352ac3bc9698/filebeat.conf

But I am leveraging the tags system where i get multiple filebeat.conf files down.
But because it starts the process with “default” --path.data & --path.logs then it can’t start another process.

I simply get:

Exiting: /var/lib/filebeat/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data)

on my 2nd config file.
My log collector config looks like this:

so i don’t know how I should change the executable params since it’s just loading config with -c %s

[drewmiranda-gl (Drew Miranda)]

I replied to your github issue : Sidecar 1.4.0 with filebeat doesn’t change path.data & path.logs · Issue #475 · Graylog2/collector-sidecar · GitHub .

I believe this is related to Cannot successful use more than one winlogbeat configuration · Issue #14228 · Graylog2/graylog2-server · GitHub which appears to be the same issue. It appears the answer so far is to update the existing config template: Cannot successful use more than one winlogbeat configuration · Issue #14228 · Graylog2/graylog2-server · GitHub

Hope that helps.

[zyxep]

it’s funny, because my graylog setup is brand new.
I tried to do migration from old collector to new, but didn’t get it to work and then i waited and got an opportunity to deploy brand new setup.

my configs does have the ${sidecar.spoolDir} in data and logs.
But clearly the runtime of filebeat (in my case) doesn’t updated variables from the config file.
I believe (without knowing filebeat), that if you don’t define --path.data for example to runtime, then filebeat will just read the paths from the config file.

But the variables are not defined anywhere in my configs, so it must come straight from go code or somewhere inside graylog server.

[drewmiranda-gl (Drew Miranda)]

I can try to reproduce. Are you running graylog 5.1.3? Can you also share a copy of your collector config you are applying to the sidecar?

And you said you are using filebeat 8.9 with graylog sidecar 1.4.0.

Thanks!

[drewmiranda-gl (Drew Miranda)]

I just did a quick test and can confirm the ${sidecar.spoolDir} variable does work:

Example sidecar config i used:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
fields.source: ${sidecar.nodeName}

filebeat.inputs:
- input_type: log
  paths:
    - /home/drew/test.log
  type: log
output.logstash:
   hosts: ["192.168.0.106:5044"]
path:
  data: ${sidecar.spoolDir}/data
  logs: ${sidecar.spoolDir}/log

I can see the correct data and log paths are used:

I can also see i have 2 unique instances of filebeat running at once (one is the test i’m doing here the other is my existing config unrelated to this test):

root@hplap:/var/lib/graylog-sidecar/generated/64c977ae0053d956a126d792# ps aux | grep filebeat
root     1788341  0.0  0.2 1440896 82684 ?       Sl   21:23   0:01 /usr/share/filebeat/bin/filebeat -c /var/lib/graylog-sidecar/generated/64c977ae0053d956a126d792/filebeat.conf
root     2319195  1.4  0.1 1658124 50964 ?       Sl   Jun29 682:48 /usr/share/filebeat/bin/filebeat -c /var/lib/graylog-sidecar/generated/64219d0478419f473d47d2b1/filebeat.conf

Data shipped to graylog as expected: