Existing sidecars disappeared, and new sidecars don't appear

1. Describe your incident:
I have just noticed that, since the beginning of December 2023, none of my Windows Servers running Sidecar 1.4.0 have been sending logs to my Graylog node. I suspect that this timing aligns with when I updated from release 5.0 to 5.2. Additionally, all of the Sidecar instances under System / Sidecars / Overview are gone. All other non-sidecar inputs are still receiving and processing messages, and if I manually run the “Graylog collector sidecar -winlogbeat-*” service found on the Windows clients, they do successfully send logs to the Beats input I have configured.

2. Describe your environment:

  • OS Information:
    Graylog node: Ubuntu Server 22.04.4 with Graylog directly installed
    Clients: Windows Server 2016, 2019
  • Package Version:
    Graylog 5.2.4
    Elasticsearch 7.10.2
    Mongodb 6.0.13
  • Service logs, configurations, and environment variables:
    Single Graylog node with one IP for web/API access and another IP for log collection. All HTTP only, no HTTPS. Typical load is 1,000-2,000 messages per second.
    32 Windows Server 2016/2019 clients running Sidecar 1.4.0 using winlogbeat

Excerpt taken from sidecar.log on a Windows client:
time=“2024-02-29T14:49:08-05:00” level=error msg=“Error fetching server version Get "htp://x.x.x.x:9000/api": dial tcp x.x.x.x:9000: connectex: No connection could be made because the target machine actively refused it."
time=“2024-02-29T14:49:39-05:00” level=error msg="Error fetching server version Get "ht
p://x.x.x.x:9000/api": dial tcp x.x.x.x:9000: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.”
time=“2024-02-29T14:54:07-05:00” level=error msg=“[UpdateRegistration] Bad response from Graylog server: 401 Unauthorized”

3. What steps have you already taken to try and solve the problem?
I have tried uninstalling and reinstalling Sidecar on the Windows clients, both the current version 1.4.0 and the new version 1.5.0 which is listed as required for Graylog 5.2, but the sidecars still do not appear under System / Sidecars. Even a fresh installation of Sidecar 1.5.0 on a new Windows client does not show up. I have also tried generating and using a new API key for both existing and new installations to no avail. I have confirmed that the Graylog API is reachable from a browser on htp://x.x.x.x:9000/api/api-browser/, htp://x.x.x.x:9000/api/sidecars/, etc.

4. How can the community help?
I’m rather new to Graylog, Linux in general, and anything programming-adjacent, so any pointers in the right direction would be appreciated. Is it an API authentication issue? And if so, what can I do about it?

Can anyone advise? I am still experiencing the issue.

Hey @so-otboe

From the log on GL-Sidecar

I see this

Looks like your missing a “T” in http://

That was a formatting error on my part, I had replaced one of the "t"s in each instance of “http” with an asterisk because the forum would not let me post more than a certain amount of links, which those counted as. It ended up eating them and italicizing some of the text instead.

Regardless, I have solved the issue, if you can call it that. I have since updated to Graylog 5.2.5 and once again generated new API tokens and uninstalled and reinstalled the Sidecars on my Windows servers, and they began working normally.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.