Graylog Sidecars intermittently losing configuration

Hello,

I have a problem with my sidecar services regularly dropping out and coming back after ~20 minutes.
I’m worried that I might be missing important events in that time, so I would really like to see this fixed.

Below is the full log of starting the service (after deleting the old logfile).

time=“2023-01-17T07:54:37+01:00” level=info msg=“Starting signal distributor”
time=“2023-01-17T07:54:48+01:00” level=info msg=“Adding process runner for: custom_servers
time=“2023-01-17T07:54:48+01:00” level=info msg=“[custom_servers] Configuration change detected, rewriting configuration file.”
time=“2023-01-17T07:54:48+01:00” level=info msg=“[custom_servers] Starting (svc driver)”
time=“2023-01-17T10:36:20+01:00” level=info msg=“Removing process runner: custom_servers
time=“2023-01-17T10:36:20+01:00” level=info msg=“[custom_servers] Stopping”
time=“2023-01-17T10:36:20+01:00” level=info msg=“Removing stale graylog service graylog-collector-custom_servers
time=“2023-01-17T10:36:20+01:00” level=info msg=“Uninstalling service graylog-collector-custom_servers
time=“2023-01-17T10:36:22+01:00” level=info msg=“No configurations assigned to this instance. Skipping configuration request.”
time=“2023-01-17T10:55:09+01:00” level=info msg=“Adding process runner for: custom_servers
time=“2023-01-17T10:55:09+01:00” level=info msg=“[custom_servers] Configuration change detected, rewriting configuration file.”
time=“2023-01-17T10:55:09+01:00” level=info msg=“[custom_servers] Starting (svc driver)”

It is clear that the configuration “custom_servers” was applied successfully detected and applied right away. After about 2½ hours the sidecar suddenly decided that this config is no longer assigned, so it removes it.
About 20 minutes later it re-applies the configuration and everything seems fine again.

This can happen a couple of times per day, or sometimes after a few days. It occurs on all servers that have Sidecar installed.
It all started since December 17 last year, after upgrading from 4.3.10 to 5.0.1. Before that, none of the sidecars had these issues.

I tried to replace the server API token with a new token, but that didn’t change anything (didn’t really expect it to).
The Graylog server logs show nothing at all.

Currently running Graylog Enterprise 5.0.2 with a free license.
All sidecars are installed on Windows Server 2016 and I’m using NXLog-CE version 2.10.2150.

Does anyone have an idea what I can do to solve this? Or should I file a bug report?

Thanks for reading!

Hey @Bobtb

From what you discribe it sound like a but, but I would not only check sidecar logs but Windows
Event-Viewer to see what else is going on. What version of Graylog sidecar are you using, was that also upgraded?

That’s a great idea, thanks for the suggestion.
The Sidecar I’m running is 1.1.0-1 and version 1.4.0-1 does seem to have improved some things.

I’ll upgrade my Sidecar instances and come back here to post the results later.

2 Likes