Messages not populating, files found in Graylog sidecars and filebeat in docker

[sashah]

1. Describe your incident:
I’m trying to ingest a log file in a docker environment using filebeat and graylog sidecar.

I’ve managed to get the filebeat client container talking to the graylog host container, the sidecar is running and if I click into the status page I can see my files. However, no messages are available in the search page.

I am using a static log file that is not receiving live updates - right now I’m trying to understand how to use graylog more then I am anything else - but I have tried adding lines to the log file and although the size gets updated no messages get added.

I am having an error in the collectors status: auditbeat Collector configuration file is not valid, waiting for the next update.
A similar error comes up when I run graylog-sidecar -debug on my client container.

However I don’t believe this is the cause of my issue as the filebeat collect is running without error. There are no other errors anywhere in the stack that I have found.

2. Describe your environment:

• OS Information:

  • I’m running docker on Windows 10.
  • I’m using two containers:
    1. The graylog host container (and the offical related containers installed using the official unmodified docker compose file)
    2. A custom made ubuntu container with filebeat and graylog-sidecar installed on it

• Package Version:

  • Graylog 5.2

• Service logs, configurations, and environment variables:

Sidecar filebeat Linux config

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
   hosts: ["${user.graylog_host}:5044"]
path:
   data: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/data
   logs: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/log

filebeat.inputs:
- input_type: log
  ignore_older: 9999h
  enabled: true
  paths:
    - /usr/share/filebeat/logs/test.log
  type: log

sidecar.yml

server_url: "http://127.0.0.1:9000/api"
server_api_token: "..."
node_id: "search-services-sidecar"
node_name: "Search Services Sidecar"
update_interval: 10
tls_skip_verify: true
list_log_files: ["usr/share/filebeat/logs/test.log"]
cache_path: "/var/cache/graylog-sidecar"
log_path: "/var/log/graylog-sidecar"
log_rotate_max_file_size: "10MiB"
log_rotate_keep_files: 10
collector_validation_timeout: "1m"
collector_shutdown_timeout: "10s"
collector_configuration_directory: "/var/lib/graylog-sidecar/generated"

tags:
  - default

collector_binaries_accesslist:
  - "/usr/bin/filebeat"
  - "/usr/lib/graylog-sidecar/filebeat"
  - "/usr/share/filebeat/bin/filebeat"

I also have a global Beats input on port 5044, but as I understand it that would factor in later down the process then I’m having problems at.

3. What steps have you already taken to try and solve the problem?
I was using the official filebeat container, but I was a little mystified on how that worked and thought the problem might be that graylog wasn’t properly running the filebeat binary so switched to a custom built one.

I have verified

• the binary file is located at /usr/share/filebeat/bin/filebeat on the client machine
• The Executable Path field in the collector is set to /usr/share/filebeat/bin/filebeat
• The Log Files section of the sidecar’s status page updates size as expected when the log file is modified

4. How can the community help?
I’m new to graylog, and not an expert in the sysadmin side of things at all so any insight into what would be causing this behavior or how I could resolve it would be very appreciated. I really just want to see the raw text of the log file’s lines in graylog, once I get that I’m confident I can move things forward again.

Thank you for any help you can give, or any similar issue you could link to.