Sending local log from Sidecar and Filebeat to Graylog

Graylog Central (peer support)
1

Good morning guys,

I’m stuck getting my log transfered to Graylog and I would love to have your help !

So far, I installed Graylog 5.1, Mongodb and Opensearch on only one instance. I also installed filebeat and graylod-sidecar.

The configuration of the collector :

image
image946×802 18.1 KB
.

The sidecar appears in status Running on the web interface.

I will post also the different configurations :

  • sidecar.yml
root@ubuntu-2204:/etc/graylog/sidecar# grep -v "^#\|^$" /etc/graylog/sidecar/sidecar.yml

server_url: "http://192.168.174.131:9000/api/"
server_api_token: "14912m0gflou74f968ro8d3d26h8cklv5d52q7btlb9lmu10bclr"
node_id: "file:/etc/graylog/sidecar/node-id"
update_interval: 10
tls_skip_verify: true
send_status: true
cache_path: "/var/cache/graylog-sidecar"
log_path: "/var/log/graylog-sidecar"
log_rotate_max_file_size: "10MiB"
log_rotate_keep_files: 10
collector_configuration_directory: "/var/lib/graylog-sidecar/generated"
collector_binaries_accesslist:
  - "/usr/share/filebeat/bin/filebeat"

The config filebeat generated :

root@ubuntu-2204:/etc/graylog/sidecar# grep -v "^#\|^$" /var/lib/graylog-sidecar/generated/654ff5f436ce7d6abdc74a4e/filebeat-mtaauth.conf

fields_under_root: true
fields.collector_node_id: ubuntu-2204
fields.gl2_source_collector: d6e10b89-c363-4afb-864b-9db8db294419
filebeat.inputs:
- input_type: log
  enabled: true
  tags:
    - mtaauth
  ssl:
   verification_mode: none
  paths:
    - /var/mail_logs/2023-11-10/*.log
  type: log
output.logstash:
   hosts: ["192.168.174.131:5044"]
path:
  data: /var/lib/graylog-sidecar/generated/654ff5f436ce7d6abdc74a4e/data
  logs: /var/lib/graylog-sidecar/generated/654ff5f436ce7d6abdc74a4e/log

When i start the graylog-sidecar service :

root@ubuntu-2204:/etc/graylog/sidecar# systemctl status graylog-sidecar

● graylog-sidecar.service - Wrapper service for Graylog controlled collector
     Loaded: loaded (/etc/systemd/system/graylog-sidecar.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-11-12 04:54:41 EST; 11min ago
   Main PID: 52009 (graylog-sidecar)
      Tasks: 17 (limit: 4583)
     Memory: 57.2M
        CPU: 1.586s
     CGroup: /system.slice/graylog-sidecar.service
             ├─52009 /usr/bin/graylog-sidecar
             └─52022 /usr/share/filebeat/bin/filebeat -c /var/lib/graylog-sidecar/generated/654ff5f436ce7d6abdc74a4e/filebeat-mtaauth.conf

Nov 12 04:54:41 ubuntu-2204 systemd[1]: Started Wrapper service for Graylog controlled collector.
Nov 12 04:54:41 ubuntu-2204 graylog-sidecar[52009]: time="2023-11-12T04:54:41-05:00" level=info msg="Using node-id: d6e10b89-c363-4afb-864b-9db8db294419"
Nov 12 04:54:41 ubuntu-2204 graylog-sidecar[52009]: time="2023-11-12T04:54:41-05:00" level=info msg="No node name was configured, falling back to hostname"
Nov 12 04:54:41 ubuntu-2204 graylog-sidecar[52009]: time="2023-11-12T04:54:41-05:00" level=info msg="Starting signal distributor"
Nov 12 04:54:41 ubuntu-2204 graylog-sidecar[52009]: time="2023-11-12T04:54:41-05:00" level=info msg="Adding process runner for: filebeat-mtaauth-654ff5f436ce7d6abdc74a4e"
Nov 12 04:54:41 ubuntu-2204 graylog-sidecar[52009]: time="2023-11-12T04:54:41-05:00" level=info msg="[filebeat-mtaauth-654ff5f436ce7d6abdc74a4e] Configuration change detected, rewriting configuration file."
Nov 12 04:54:42 ubuntu-2204 graylog-sidecar[52009]: time="2023-11-12T04:54:42-05:00" level=info msg="[filebeat-mtaauth-654ff5f436ce7d6abdc74a4e] Starting (exec driver)"

Nothing wrong when I check the graylog-sidecar.log :

root@ubuntu-2204:/etc/graylog/sidecar# cat /var/log/graylog-sidecar/sidecar.log

time="2023-11-12T05:10:27-05:00" level=info msg="Starting signal distributor"
time="2023-11-12T05:10:27-05:00" level=info msg="Adding process runner for: filebeat-mtaauth-654ff5f436ce7d6abdc74a4e"
time="2023-11-12T05:10:27-05:00" level=info msg="[filebeat-mtaauth-654ff5f436ce7d6abdc74a4e] Configuration change detected, rewriting configuration file."
time="2023-11-12T05:10:27-05:00" level=info msg="[filebeat-mtaauth-654ff5f436ce7d6abdc74a4e] Starting (exec driver)"
time="2023-11-12T05:13:09-05:00" level=info msg="Stopping signal distributor"
time="2023-11-12T05:13:09-05:00" level=info msg="[filebeat-mtaauth-654ff5f436ce7d6abdc74a4e] Stopping"
time="2023-11-12T05:13:09-05:00" level=info msg="Starting signal distributor"
time="2023-11-12T05:13:09-05:00" level=info msg="Adding process runner for: filebeat-mtaauth-654ff5f436ce7d6abdc74a4e"
time="2023-11-12T05:13:09-05:00" level=info msg="[filebeat-mtaauth-654ff5f436ce7d6abdc74a4e] Configuration change detected, rewriting configuration file."
time="2023-11-12T05:13:09-05:00" level=info msg="[filebeat-mtaauth-654ff5f436ce7d6abdc74a4e] Starting (exec driver)"

Wish you a great day and thanks for your help !

2

Hey @Okizeme

To be honest I really dont see anything wrong. Can you discribe you issue?

3

Hi gsmith,

Thx for ur message, I finally managed to make it work.

Here the modified conf in case it can help someone :

# Needed for Graylog
fields_under_root: true
fields:
  collector_node_id: ${sidecar.nodeName}
  gl2_source_collector: ${sidecar.nodeId}
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /root/mail_logs/*.log

output.logstash:
  hosts: ["10.123.8.112:5044"]
path:
  data: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/data
  logs: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/log
ssl:
  verification_mode: none

Best wishes,

Pierre

4

hey

Awesome, thanks for the feed back @Okizeme :+1:

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.