Graylog - Unauthorized errors in the log


I have 3 Graylog nodes running behind an Oauth2 proxy and a Loadbalancer. My setup is like

user -->> LB --> Nginx -->> Oauth2_proxy —>> Graylog Clsuter.

I get lots of following warning in the log

Oct 07 20:10:00 node01.graylog. docker[2616115]: 2020-10-07 18:10:00,421 WARN : - Unable to call http://private-ip-of-the-vm:9000/api/system on node , result: Unauthorized
Oct 07 20:10:02 node01.graylog. docker[2616115]: 2020-10-07 18:10:02,151 WARN : - Unable to call http://private-ip-of-the-vm:9000/api/system/metrics/multiple on node , result: Unauthorized

I am running Graylog version 3.3.6 in Docker with the following config

GRAYLOG_SERVER_JAVA_OPTS=\"-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Xms6g -Xmx6g\""

I am also getting the following error in the web interface while accessing thee nodes and all.

Could not get JVM information
Getting JVM information for node ‘d686b638-3175-4d61-b529-6d4e0025bca1’ failed: Error: cannot GET https://Public-URL/api/cluster/d686b638-3175-4d61-b529-6d4e0025bca1/jvm (502)

If I try curl from the host machine, it works

curl -i “
HTTP/1.1 200 OK
X-Graylog-Node-ID: d686b638-3175-4d61-b529-6d4e0025bca1
X-Runtime-Microseconds: 3573
Content-Type: application/json
Date: Wed, 07 Oct 2020 18:23:54 GMT
Content-Length: 253
“cluster_id” : “73031db4-cf74-4e2d-a7bd-126f923dc3cf”,
“node_id” : “d686b638-3175-4d61-b529-6d4e0025bca1”,
“version” : “3.3.6+92fb41e”,
“tagline” : “Manage your logs in the dark and have lasers going and make it look like you’re from space!”

I am running Graylog with almost same settings (no proxy and Oauth2_proxy) in k8s without any issues.

Any help would be highly appreciated.

Arun Sasidharan

Check your Configure Provider Order
in System / Authentication / Configure Provider Order

I have following order


Try to elaborate with order, if it helps, it’s only guess.

Sorry @shoothub, I didn’t understand your last comment.

I changed the Authentication provider order to enable SSO with Azure AD. I am using SSO plugin version 3.3.0.

I mean to change order of providers, maybe it helps.

I tried the following order of providers. But still the same error.

Try #1

Try #2

Hello there,

Any suggestions?

Thanks in advance,

Hey Arun, I’m little new to K8s env. is possible to share your the steps to run your setup with Oauth2_proxy? Thanks