1. Describe your incident:
I have just noticed that, since the beginning of December 2023, none of my Windows Servers running Sidecar 1.4.0 have been sending logs to my Graylog node. I suspect that this timing aligns with when I updated from release 5.0 to 5.2. Additionally, all of the Sidecar instances under System / Sidecars / Overview are gone. All other non-sidecar inputs are still receiving and processing messages, and if I manually run the “Graylog collector sidecar -winlogbeat-*” service found on the Windows clients, they do successfully send logs to the Beats input I have configured.
2. Describe your environment:
- OS Information:
Graylog node: Ubuntu Server 22.04.4 with Graylog directly installed
Clients: Windows Server 2016, 2019 - Package Version:
Graylog 5.2.4
Elasticsearch 7.10.2
Mongodb 6.0.13 - Service logs, configurations, and environment variables:
Single Graylog node with one IP for web/API access and another IP for log collection. All HTTP only, no HTTPS. Typical load is 1,000-2,000 messages per second.
32 Windows Server 2016/2019 clients running Sidecar 1.4.0 using winlogbeat
Excerpt taken from sidecar.log on a Windows client:
time=“2024-02-29T14:49:08-05:00” level=error msg=“Error fetching server version Get "htp://x.x.x.x:9000/api": dial tcp x.x.x.x:9000: connectex: No connection could be made because the target machine actively refused it."
time=“2024-02-29T14:49:39-05:00” level=error msg="Error fetching server version Get "htp://x.x.x.x:9000/api": dial tcp x.x.x.x:9000: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.”
time=“2024-02-29T14:54:07-05:00” level=error msg=“[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized”
3. What steps have you already taken to try and solve the problem?
I have tried uninstalling and reinstalling Sidecar on the Windows clients, both the current version 1.4.0 and the new version 1.5.0 which is listed as required for Graylog 5.2, but the sidecars still do not appear under System / Sidecars. Even a fresh installation of Sidecar 1.5.0 on a new Windows client does not show up. I have also tried generating and using a new API key for both existing and new installations to no avail. I have confirmed that the Graylog API is reachable from a browser on htp://x.x.x.x:9000/api/api-browser/, htp://x.x.x.x:9000/api/sidecars/, etc.
4. How can the community help?
I’m rather new to Graylog, Linux in general, and anything programming-adjacent, so any pointers in the right direction would be appreciated. Is it an API authentication issue? And if so, what can I do about it?