Hello,
I have a problem with my sidecar services regularly dropping out and coming back after ~20 minutes.
I’m worried that I might be missing important events in that time, so I would really like to see this fixed.
Below is the full log of starting the service (after deleting the old logfile).
time=“2023-01-17T07:54:37+01:00” level=info msg=“Starting signal distributor”
time=“2023-01-17T07:54:48+01:00” level=info msg=“Adding process runner for: custom_servers”
time=“2023-01-17T07:54:48+01:00” level=info msg=“[custom_servers] Configuration change detected, rewriting configuration file.”
time=“2023-01-17T07:54:48+01:00” level=info msg=“[custom_servers] Starting (svc driver)”
time=“2023-01-17T10:36:20+01:00” level=info msg=“Removing process runner: custom_servers”
time=“2023-01-17T10:36:20+01:00” level=info msg=“[custom_servers] Stopping”
time=“2023-01-17T10:36:20+01:00” level=info msg=“Removing stale graylog service graylog-collector-custom_servers”
time=“2023-01-17T10:36:20+01:00” level=info msg=“Uninstalling service graylog-collector-custom_servers”
time=“2023-01-17T10:36:22+01:00” level=info msg=“No configurations assigned to this instance. Skipping configuration request.”
time=“2023-01-17T10:55:09+01:00” level=info msg=“Adding process runner for: custom_servers”
time=“2023-01-17T10:55:09+01:00” level=info msg=“[custom_servers] Configuration change detected, rewriting configuration file.”
time=“2023-01-17T10:55:09+01:00” level=info msg=“[custom_servers] Starting (svc driver)”
It is clear that the configuration “custom_servers” was applied successfully detected and applied right away. After about 2½ hours the sidecar suddenly decided that this config is no longer assigned, so it removes it.
About 20 minutes later it re-applies the configuration and everything seems fine again.
This can happen a couple of times per day, or sometimes after a few days. It occurs on all servers that have Sidecar installed.
It all started since December 17 last year, after upgrading from 4.3.10 to 5.0.1. Before that, none of the sidecars had these issues.
I tried to replace the server API token with a new token, but that didn’t change anything (didn’t really expect it to).
The Graylog server logs show nothing at all.
Currently running Graylog Enterprise 5.0.2 with a free license.
All sidecars are installed on Windows Server 2016 and I’m using NXLog-CE version 2.10.2150.
Does anyone have an idea what I can do to solve this? Or should I file a bug report?
Thanks for reading!