Filebeat and linux messages file

i just installed the graylog appliance and a rhel7 test client.
i created a filebeats input and installed sidecar on the client.
the client is showing up and appears to be working as it is processing the logs in /var/log
if i click on the sidecar under system/sidecars, i can see all the log files it is reading and
/var/log/messages is one of them… my question is , why dont i see any messages in graylog, from
the messages file from the client…?
there is info being collected from the client, for example /var/log/yum.log.

the messages file line under the sidecar looks like this

Mayby post your sidecar configuration? Do you see anything in the sidecar logs on the client?

here is the config file

cat sidecar.yml

The URL to the Graylog server API.

server_url: “

The API token to use to authenticate against the Graylog server API.

This field is mandatory

server_api_token: “hidden”

The node ID of the sidecar. This can be a path to a file or an ID string.

If set to a file and the file doesn’t exist, the sidecar will generate an

unique ID and writes it to the configured path.

Example file path: “file:/etc/graylog/sidecar/node-id”

Example ID string: “6033137e-d56b-47fc-9762-cd699c11a5a9”

ATTENTION: Every sidecar instance needs a unique ID!

node_id: “file:/etc/graylog/sidecar/node-id”

The node name of the sidecar. If this is empty, the sidecar will use the

hostname of the host it is running on.

#node_name: “”

The update interval in seconds. This configures how often the sidecar will

contact the Graylog server for keep-alive and configuration update requests.

#update_interval: 10

This configures if the sidecar should skip the verification of TLS connections.

Default: false

#tls_skip_verify: false

This enables/disables the transmission of detailed sidecar information like

collector statues, metrics and log file lists. It can be disabled to reduce

load on the Graylog server if needed. (disables some features in the server UI)

#send_status: true

A list of directories to scan for log files. The sidecar will scan each

directory for log files and submits them to the server on each update.



- “/var/log/nginx”

- “/opt/app/logs”

Default: empty list

list_log_files: [/var/log]

Directory where the sidecar stores internal data.

#cache_path: “/var/cache/graylog-sidecar”

Directory where the sidecar stores logs for collectors and the sidecar itself.

#log_path: “/var/log/graylog-sidecar”

The maximum size of the log file before it gets rotated.

#log_rotate_max_file_size: “10MiB”

The maximum number of old log files to retain.

#log_rotate_keep_files: 10

Directory where the sidecar generates configurations for collectors.

#collector_configuration_directory: “/var/lib/graylog-sidecar/generated”

A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the whitelist feature.

Wildcards can be used, for a full pattern description see



- “/usr/bin/filebeat”

- “/opt/collectors/*”

Example disable whitelisting:




- “/usr/bin/filebeat”

- “/usr/bin/packetbeat”

- “/usr/bin/metricbeat”

- “/usr/bin/heartbeat”

- “/usr/bin/auditbeat”

- “/usr/bin/journalbeat”

- “/usr/share/filebeat/bin/filebeat”

- “/usr/share/packetbeat/bin/packetbeat”

- “/usr/share/metricbeat/bin/metricbeat”

- “/usr/share/heartbeat/bin/heartbeat”

- “/usr/share/auditbeat/bin/auditbeat”

- “/usr/share/journalbeat/bin/journalbeat”

- “/usr/bin/nxlog”

- “/opt/nxlog/bin/nxlog”

i dont see anything in /var/log/graylog-sidecar/sidecar.log

all i see is this

time=“2020-08-10T17:10:21-04:00” level=info msg=“Adding process runner for: filebeat”
time=“2020-08-10T17:10:21-04:00” level=info msg="[filebeat] Configuration change detected, rewriting configuration file."
time=“2020-08-10T17:10:21-04:00” level=info msg="[filebeat] Starting (exec driver)"
time=“2020-08-11T08:51:33-04:00” level=info msg=“Stopping signal distributor”
time=“2020-08-11T08:51:33-04:00” level=info msg="[filebeat] Stopping"
time=“2020-08-11T08:51:33-04:00” level=info msg=“Starting signal distributor”

its just strange that i see all the logs it claims to be reading but if i search for filebeat_log_file_path:/var/log/messages i get nothing…

i think i found it. it appears to be permissions for the files in ./var/log

for ex… filebeat_log_file_path:/var/log/Xorg.0.log is 644
but messages is 400… it seems that anything 400 i cant search on… but yet in the sidecar view, it shows all the files in /var/log as intake…
so im not sure why i cant search them

scratch that…
that didnt seem to matter… but perhaps its only files that end with .log that seem to be readable…
because /var/log/dmesg is in the list for this sidecar and the permissions are 644 but i cant search it…
but i can search vmware-network.log and that is 400

First - please use the forum tools to make your yml file easier to read.

For instance using the preformatted text tool AND removing all the standard comments that are not needed your .yml boils down to the following… which is much more readable for troubleshooting.

server_url: “”
server_api_token: “hidden”
node_id: “file:/etc/graylog/sidecar/node-id”
list_log_files: [/var/log]

But what I am interested is in the sidecar configuration. The yml file sets the framework for the sidecar but it is the sidecars->configuration that you have applied to the client that defines the specifics of what messages to send into Graylog. Here is a working configuration I have applied to a linux client: (formatted nicely of course:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

- type: log
  enabled: true
  exclude_lines: ['cron\:session']
    - /var/log/auth.log
  ignore_older: 72h
    - linux
#############   field marker for shunting with stream rules
    beats_os: linux

   - ${user.BeatsInput}

  data: /var/cache/graylog-sidecar/filebeat/data
  logs: /var/log/graylog-sidecar

sorry about that…
so if i go to sidecars and select the car, then configuration it says…

 #Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

- input_type: log
- /var/log/*.log
  type: log
   hosts: [""]
  data: /var/lib/graylog-sidecar/collectors/filebeat/data
  logs: /var/lib/graylog-sidecar/collectors/filebeat/log

My first guess is indentation - which is meaningful in YAML files. Make sure the path you want is indented… with (specifically) two spaces, and don’t use tabs…

  - input_type: log
      - /var/log/*.log
    type: log

Here are some elastic tips:

the indentations are there…
not sure why it didnt carry over when i pasted it…

im not understanding why all the log files show up under the sidecar but when i search for them, i cant find some of them… for example /var/log/messages and var/log/audit.log are listed but if I do filebeat_source:/var/log/audit.log it returns nothing and if i do filebeat_source:/var/log/boot.log it returns everything in that logfile…

Not sure… either try searching fro everything that is NOT filebeat_source:/var/log/boot.log

i changed the “paths:” in the filebeat config to say this.
- /var/log/.log
- /var/log/audit/
- /var/log/messages

now i am getting audits and messages…
still dont understand why all of /var/log shows up under the sidecar when you click it…
its like an overview of what it claims to be collecting… but yet i cant see all of it…
for example, it claims to have /var/log/tallylog but if i search for it, its not there…
possibly because the config file says /var/log/*.log? and tallylog does not have a .log at the end of it ?

Certainly *.log will only snag files that end in .log… which would mean you would not pick up /var/log/messages. If you want to go into all subdirectories you I think it would be something like /var/log/*/*.log but you would still need a separate line in your sidecar configuration for messages.

thanks for your help on this !

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.