Sidecars Have Gone Missing?

[accidentaladmin (Matthew)]

1. Describe your incident:

All my Sidecars have gone missing within the Sidecars Overview/Administration sections (but they are still running and collecting data)

2. Describe your environment:

• OS Information:
  All environments are Debian 12 within a LXD container hosted by PVE
• Package Version:
  Graylog 5.1.7
  Mongodb 6.0.10
  opensearch 2.9.0
• Service logs, configurations, and environment variables:

My setup:

My missing sidecars:

Buuuttttt…

They are still working.

Any ideas?

[Joel_Duffield]

So your agents are running, its sidecar with the issue. I would check the config file of a sidecar itself, make sure it has a valid url and that you can reach the graylog api from that machine. Also check that the api key its using is still valid.

Sidecar makes api calls to the graylog server to check in, if its not checking in after some inactive time the record of it will be cleaned up.

[accidentaladmin (Matthew)]

Ooof. This is not going to be fun. What I don’t understand is how they ALL disappeared at the same time. Its not like they all have the same api key

[Joel_Duffield]

You generated a different api key for each one? My first guess would be that thr api just isnt reachable from those machines anymore for some reason.

[accidentaladmin (Matthew)]

Yup; woke up with that same conclusion and verified.

Around 7 days ago I configured Cloudflare as a gatekeeper (forcing Azure [Entra?] AD as MFA). Disabling that resolved the issue. So simple, I forgot.

I wonder if there is a way to accept API keys