Failed to report collector status to server

Graylog Central (peer support)
, ,
1

I’m in the process of switching from Collector Sidecars (Legacy) to Sidecars and hitting an issue with the Linux sidecar client reporting:

graylog-sidecar: time="2021-03-29T11:14:52-04:00" level=error msg="[UpdateRegistration] Failed to report collector status to server: Put \"https://graylog.company.com/api/sidecars/5...nodeid...3\": EOF"

The client is checking in and connecting and I can see it under Sidecars Overview. It’s showing the stats and picking up config changes. But it appears to be failing intermittently. Defaults are being used for send_status, update_interval, etc…

Strangely I’m not seeing this issue on the Windows client with similar config.

With the default 10 second interval I’m seeing these random error times:
time=“2021-03-29T11:18:03-04:00” level=error…
time=“2021-03-29T11:18:43-04:00” level=error…
time=“2021-03-29T11:19:23-04:00” level=error…
time=“2021-03-29T11:20:33-04:00” level=error…
time=“2021-03-29T11:21:43-04:00” level=error…
time=“2021-03-29T11:22:33-04:00” level=error…
time=“2021-03-29T11:25:04-04:00” level=error…
time=“2021-03-29T11:25:44-04:00” level=error…
time=“2021-03-29T11:26:04-04:00” level=error…
time=“2021-03-29T11:27:34-04:00” level=error…

I’ve tried tweaking the sidecar.yml on the Linux client but that didn’t help.

The URL to the Graylog server API.

server_url: “https://graylog.company.com/api/”
#server_url: “https://graylog.company.com:443/api/”
#server_url: “https://graylog.company.com:443/”
#server_url: “https://graylog.company.com/”

Setup:
Graylog 4.0.5 - 3 node cluster
Haproxy load balancing to Nginx.
Nginx proxying to local 9000 on the host
All Centos 7

Sidecar on Centos7
graylog-sidecar-1.1.0-1.x86_64
filebeat-7.12.0-1.x86_64

Sidecar on WIndows
1.1.0

Any ideas? Or if this is a known issue? I checked Github but didn’t find it.

2

Just found this under collector-sidecars github

3

look in the logs of the linux client /var/log/graylog-sidecar There is a filebeat log and a sidecar.log file that may give you more information on what is happening. You could post your sidecar config which might help for diagnostic…

Also use the formatting tools </> for code posts… it make it easier for people to read:
image

1 Like
4

I didn’t post any code but updated it with the formatting for the long log entry at the top. Taking a look at the logs now.

Edit: sidecar.log is showing the same entries with no additional information. This may be the bug I posted above.

filebeat_stderr.log has no entries.

5

Same issue here:

6

Looks like one of those (older) posts has nxlog and filebeat enabled - random guess… if your setup looked like that and you weren’t using one of those, you could try disabling it. maybe the enabled one is trying to report something you aren’t using to the server?

7

Just have filebeat on and that’s working correctly. The issue appears to be the sidecar reporting stats and randomly failing.

8

Maybe run sidecar in debug mode to see which Graylog server its having a hard time reporting to:

graylog-sidecar -debug

9

Reports the same information. The sidecar doesn’t appear to see which server it’s talking to as it’s load balanced before hitting the server.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.