Linux sidecar checks logs it's not configured for

Graylog Central (peer support)
1

I have a Linux sidecar and running a filebeat collector on RHEL9, which is configured only to check /var/log/secure:

filebeat.inputs:

- type: filestream
  id: log-filestream
  enabled: true
  paths:
  - /var/log/secure
  fields:
      event_source_product: rsyslog

I recently updated the graylog-sidecar service to run as an unprivileged user, updated ownership of /var/lib/graylog-sidecar and /etc/graylog, and used an ACL to grant read permission on /var/log/secure since it has root-only permissions. Everything is working as expect, I’m getting the messages from /var/log/secure in Graylog, but in the sidecar log I’m now seeing messages like these, every 10 seconds:

time="2026-02-05T09:36:26-05:00" level=error msg="Can not get file list for /var/log/audit: open /var/log/audit: permission denied"
time="2026-02-05T09:36:26-05:00" level=error msg="Can not get file list for /var/log/chrony: open /var/log/chrony: permission denied"
time="2026-02-05T09:36:26-05:00" level=error msg="Can not get file list for /var/log/insights-client: open /var/log/insights-client: permission denied"
time="2026-02-05T09:36:26-05:00" level=error msg="Can not get file list for /var/log/private: open /var/log/private: permission denied"
time="2026-02-05T09:36:26-05:00" level=error msg="Can not get file list for /var/log/sssd: open /var/log/sssd: permission denied"

I can’t find any reason for the sidecar service to be doing anything with these files. They’re not in any config I have checked, nor do most of them even exist on the host. I’d like to fix whatever is causing this, and stop the log spam this issue is causing.

Any help is greatly appreciated! Thanks!

2

Actually, these paths do exist on the host, but they’re directories which the runtime user does not have access to. Using strace I was able to determine that the graylog-sidecar parent process is causing this, and not the filebeat collector. It looks like graylog-sidecar is trying to walk the entire /var/log directory tree, but I can’t figure out why. It’s not trying to ingest any files, but it’s trying to traverse all of the subdirectories to see their contents.

Here are all of the calls involving /var/log in the trace of the running sidecar process. I’m stumped as to why it’s doing this when it’s only configured to check one specific log:

638325 openat(AT_FDCWD, "/var/log/", O_RDONLY|O_CLOEXEC) = 7
638325 newfstatat(AT_FDCWD, "/var/log/README", {st_mode=S_IFLNK|0777, st_size=39, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/anaconda",  <unfinished ...>
638325 openat(AT_FDCWD, "/var/log/anaconda", O_RDONLY|O_CLOEXEC) = 7
638325 newfstatat(AT_FDCWD, "/var/log/anaconda/X.log", {st_mode=S_IFREG|0600, st_size=17581, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/anaconda/anaconda.log", {st_mode=S_IFREG|0600, st_size=88189, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/anaconda/dbus.log",  <unfinished ...>
638325 newfstatat(AT_FDCWD, "/var/log/anaconda/dnf.librepo.log", {st_mode=S_IFREG|0600, st_size=66156, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/anaconda/hawkey.log", {st_mode=S_IFREG|0600, st_size=180, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/anaconda/journal.log", {st_mode=S_IFREG|0600, st_size=3835607, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/anaconda/ks-script-4juoertn.log", {st_mode=S_IFREG|0600, st_size=63, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/anaconda/ks-script-_t6a9iea.log", {st_mode=S_IFREG|0600, st_size=40, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/anaconda/ks-script-o0wkauej.log", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/anaconda/lorax-packages.log", {st_mode=S_IFREG|0644, st_size=27098, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/anaconda/lvm.log", {st_mode=S_IFREG|0600, st_size=65132, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/anaconda/packaging.log",  <unfinished ...>
638325 newfstatat(AT_FDCWD, "/var/log/anaconda/program.log", {st_mode=S_IFREG|0600, st_size=15001, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/anaconda/storage.log", {st_mode=S_IFREG|0600, st_size=565106, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/anaconda/syslog",  <unfinished ...>
638325 newfstatat(AT_FDCWD, "/var/log/audit", {st_mode=S_IFDIR|0700, st_size=99, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 openat(AT_FDCWD, "/var/log/audit", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
638325 newfstatat(AT_FDCWD, "/var/log/btmp",  <unfinished ...>
638325 newfstatat(AT_FDCWD, "/var/log/btmp-20260201", {st_mode=S_IFREG|0660, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/chrony", {st_mode=S_IFDIR|0750, st_size=6, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 openat(AT_FDCWD, "/var/log/chrony", O_RDONLY|O_CLOEXEC <unfinished ...>
638325 newfstatat(AT_FDCWD, "/var/log/cron", {st_mode=S_IFREG|0600, st_size=323697, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/cron-20260111", {st_mode=S_IFREG|0600, st_size=511029, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/cron-20260118", {st_mode=S_IFREG|0600, st_size=511271, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/cron-20260125",  <unfinished ...>
638325 newfstatat(AT_FDCWD, "/var/log/cron-20260201", {st_mode=S_IFREG|0600, st_size=510925, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/dnf.librepo.log", {st_mode=S_IFREG|0644, st_size=89189, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/dnf.librepo.log.1",  <unfinished ...>
638325 newfstatat(AT_FDCWD, "/var/log/dnf.librepo.log.2", {st_mode=S_IFREG|0644, st_size=1048436, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/dnf.librepo.log.3", {st_mode=S_IFREG|0644, st_size=1048410, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/dnf.librepo.log.4", {st_mode=S_IFREG|0644, st_size=1048362, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/dnf.log",  <unfinished ...>
638325 newfstatat(AT_FDCWD, "/var/log/dnf.log.1", {st_mode=S_IFREG|0644, st_size=1048573, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/dnf.log.2", {st_mode=S_IFREG|0644, st_size=1048519, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/dnf.log.3", {st_mode=S_IFREG|0644, st_size=1048517, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/dnf.log.4",  <unfinished ...>
638325 newfstatat(AT_FDCWD, "/var/log/dnf.rpm.log", {st_mode=S_IFREG|0644, st_size=828740, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/firewalld", {st_mode=S_IFREG|0640, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/graylog-sidecar", {st_mode=S_IFDIR|0755, st_size=25, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 openat(AT_FDCWD, "/var/log/graylog-sidecar", O_RDONLY|O_CLOEXEC) = 7
638325 newfstatat(AT_FDCWD, "/var/log/graylog-sidecar/sidecar.log", {st_mode=S_IFREG|0644, st_size=4453132, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/hawkey.log", {st_mode=S_IFREG|0644, st_size=3960, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/hawkey.log-20260111", {st_mode=S_IFREG|0644, st_size=5280, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/hawkey.log-20260118", {st_mode=S_IFREG|0644, st_size=5160, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/hawkey.log-20260125", {st_mode=S_IFREG|0644, st_size=5160, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/hawkey.log-20260201",  <unfinished ...>
638325 newfstatat(AT_FDCWD, "/var/log/insights-client", {st_mode=S_IFDIR|0700, st_size=33, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 openat(AT_FDCWD, "/var/log/insights-client", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
638325 newfstatat(AT_FDCWD, "/var/log/kdump.log", {st_mode=S_IFREG|0600, st_size=3390, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/lastlog", {st_mode=S_IFREG|0664, st_size=293460, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/maillog", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/maillog-20260111", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/maillog-20260118", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/maillog-20260125", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/maillog-20260201", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/messages", {st_mode=S_IFREG|0600, st_size=11249651, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/messages-20260111", {st_mode=S_IFREG|0600, st_size=7866302, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/messages-20260118", {st_mode=S_IFREG|0600, st_size=7818325, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/messages-20260125", {st_mode=S_IFREG|0600, st_size=7803078, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/messages-20260201", {st_mode=S_IFREG|0600, st_size=7784937, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/pcp", {st_mode=S_IFDIR|0775, st_size=75, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 openat(AT_FDCWD, "/var/log/pcp", O_RDONLY|O_CLOEXEC) = 7
638325 newfstatat(AT_FDCWD, "/var/log/pcp/pmcd", {st_mode=S_IFDIR|0775, st_size=6, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 openat(AT_FDCWD, "/var/log/pcp/pmcd", O_RDONLY|O_CLOEXEC) = 7
638325 newfstatat(AT_FDCWD, "/var/log/pcp/pmfind", {st_mode=S_IFDIR|0775, st_size=6, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 openat(AT_FDCWD, "/var/log/pcp/pmfind", O_RDONLY|O_CLOEXEC) = 7
638325 newfstatat(AT_FDCWD, "/var/log/pcp/pmie", {st_mode=S_IFDIR|0775, st_size=6, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 openat(AT_FDCWD, "/var/log/pcp/pmie", O_RDONLY|O_CLOEXEC) = 7
638325 newfstatat(AT_FDCWD, "/var/log/pcp/pmlogger", {st_mode=S_IFDIR|0775, st_size=26, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 openat(AT_FDCWD, "/var/log/pcp/pmlogger", O_RDONLY|O_CLOEXEC <unfinished ...>
638325 newfstatat(AT_FDCWD, "/var/log/pcp/pmlogger/.NeedRewrite", {st_mode=S_IFREG|0644, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/pcp/pmproxy", {st_mode=S_IFDIR|0775, st_size=6, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 openat(AT_FDCWD, "/var/log/pcp/pmproxy", O_RDONLY|O_CLOEXEC) = 7
638325 newfstatat(AT_FDCWD, "/var/log/private", {st_mode=S_IFDIR|0700, st_size=6, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 openat(AT_FDCWD, "/var/log/private", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
638325 newfstatat(AT_FDCWD, "/var/log/rhsm", {st_mode=S_IFDIR|0755, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 openat(AT_FDCWD, "/var/log/rhsm", O_RDONLY|O_CLOEXEC) = 7
638325 newfstatat(AT_FDCWD, "/var/log/rhsm/rhsm.log", {st_mode=S_IFREG|0644, st_size=9903, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/rhsm/rhsm.log-20260111", {st_mode=S_IFREG|0644, st_size=14553, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/rhsm/rhsm.log-20260118", {st_mode=S_IFREG|0644, st_size=14684, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/rhsm/rhsm.log-20260125", {st_mode=S_IFREG|0644, st_size=14553, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/rhsm/rhsm.log-20260201", {st_mode=S_IFREG|0644, st_size=19253, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/rhsm/rhsmcertd.log", {st_mode=S_IFREG|0644, st_size=2082, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/rhsm/rhsmcertd.log-20260111", {st_mode=S_IFREG|0644, st_size=3290, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/rhsm/rhsmcertd.log-20260118", {st_mode=S_IFREG|0644, st_size=3290, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/rhsm/rhsmcertd.log-20260125", {st_mode=S_IFREG|0644, st_size=3290, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/rhsm/rhsmcertd.log-20260201", {st_mode=S_IFREG|0644, st_size=3290, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/secure", {st_mode=S_IFREG|0640, st_size=1391142, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/secure-20260111", {st_mode=S_IFREG|0600, st_size=3148534, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/secure-20260118", {st_mode=S_IFREG|0600, st_size=2249236, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/secure-20260125", {st_mode=S_IFREG|0600, st_size=2211440, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/secure-20260201", {st_mode=S_IFREG|0600, st_size=1885134, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/spooler", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/spooler-20260111", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/spooler-20260118", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/spooler-20260125", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/spooler-20260201", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/sssd", {st_mode=S_IFDIR|0750, st_size=6, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 openat(AT_FDCWD, "/var/log/sssd", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
638325 newfstatat(AT_FDCWD, "/var/log/tallylog", {st_mode=S_IFREG|0600, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/vmware-network.log", {st_mode=S_IFREG|0644, st_size=195, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/vmware-vgauthsvc.log.0", {st_mode=S_IFREG|0600, st_size=7917, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/vmware-vmsvc-root.log", {st_mode=S_IFREG|0600, st_size=71150, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/vmware-vmtoolsd-root.log", {st_mode=S_IFREG|0600, st_size=2845, ...}, AT_SYMLINK_NOFOLLOW) = 0
638325 newfstatat(AT_FDCWD, "/var/log/wtmp", {st_mode=S_IFREG|0664, st_size=45696, ...}, AT_SYMLINK_NOFOLLOW) = 0
3

Hey @d_a_parker,

Could you please post an example of the sidercar conf?

Below is some config which could be related.

# A list of directories to scan for log files. The sidecar will scan each
# directory for log files and submits them to the server on each update.
#
# Example:
#     list_log_files:
#       - "/var/log/nginx"
#       - "/opt/app/logs"
#
# Default: empty list
#list_log_files: []
4

Thanks @Wine_Merchant! Looks like that’s the problem:

# A list of directories to scan for log files. The sidecar will scan each
# directory for log files and submits them to the server on each update.
#
# Example:
#     list_log_files:
#       - "/var/log/nginx"
#       - "/opt/app/logs"
#
# Default: empty list
#list_log_files: []
list_log_files:
  - /var/log/

I’m not sure if that’s something I set or just the default. Either way, I’ll unset that from /var/log/ because we don’t need it scanning the whole directory.

I appreciate the help!

1 Like