[ProxiedResource] Unable to call https ... Unauthorized

[Barbocer (Alexandre Beauclair)]

Hello!

I currently have a 3-node Graylog cluster which is seemingly running fine. I have access to the data sent to my outputs, I can view metrics related to my nodes, etc.

However, in the Graylog logs there are a lot of the following messages, but only on my Master node:

WARN [ProxiedResource] Unable to call https://<SlaveNode1_IP>:9000/api/system/metrics/multiple on node <SlaveNode1_ID>, result: Unauthorized
WARN [ProxiedResource] Unable to call https://<SlaveNode2_IP>:9000/api/system/metrics/multiple on node <SlaveNode2_ID>, result: Unauthorized

Now obviously this seems like an authentication error. When running curl from the command line, here are the results:

root@MASTER:~# curl -i https://<SlaveNode1_IP>:9000/api/?pretty=true
HTTP/1.1 200 OK
X-Graylog-Node-ID: <SlaveNode1_ID>
X-Runtime-Microseconds: 931
Content-Type: application/json
Date: Tue, 12 Nov 2019 19:53:29 GMT
Content-Length: 253

{
“cluster_id” : ClusterID,
“node_id” : NodeID,
“version” : “3.1.0+aa5175e”,
“tagline” : “Manage your logs in the dark and have lasers going and make it look like you’re from >space!”
}

root@MASTER:~# curl -i https://<SlaveNode1_IP>:9000/api/system
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm=“Graylog Server”
X-Graylog-Node-ID: <SlaveNode1_ID>
X-Runtime-Microseconds: 249
Date: Tue, 12 Nov 2019 19:17:53 GMT
Content-Length: 0

As we can see, it seems to be able to connect just fine to /api, but when attempting to connect to /api/system it’s getting an “Unauthorized” error. I also attempted to use the API Browser (System → Nodes → API Browser) on the Slave nodes, and if requesting a response from /system/metrics, I get a popup asking for authentication, which matches the behavior from calling /api/system from curl.

Now although everything seems to be working nonetheless, I’m worried this might hide a problem down the line. I have nothing apart from the mentioned line in the logs, so it doesn’t tell me what it’s trying to fetch and for what purpose.

Is there an authentication option I might be missing somewhere? What is considered a “ProxiedResource”?

Additonal Information:
Graylog Version: 3.1.0
ElasticSearch Version: 6.8.3
MongoDB Version: 4.0.12

[jan (Jan Doberstein)]

Do you have any kind of proxy running in front of Graylog?

In addition I would update to the latest bugfix release of 3.1 what is 3.1.3 currently.

[Barbocer (Alexandre Beauclair)]

Hi Jan,

Yes, we have an Haproxy load balancer running in front, but it only serves to load balance the requests and the data that is being sent over to the cluster. The Graylog nodes can be reached on their own IP addresses/port if desired, so it’s not so much “in front” and blocking any traffic to individual nodes as just being there to ensure redundancy and high availability access from a single address. The Haproxy servers are also on their own, and do not run on the same servers as Graylog, so apart from forwarding traffic to the nodes, they don’t interact with the Graylog cluster whatsoever.

I will also update the cluster once I get a chance. Would the bugfixes implemented in the releases since contain something that might fix my problem specifically by any chance?

Thanks!

[jan (Jan Doberstein)]

do you have a proxy to reach the internet configured in your servers?

[Barbocer (Alexandre Beauclair)]

No, the Graylog nodes nor the load balancer use a proxy to reach the Internet. Does using a proxy usually give these sorts of messages? I don’t have any sort of authentication setup on either Load Balancers or Graylog nodes outside the LDAP and Internal user logins for the Web Interface. I also use SSL everywhere.

I’m still trying to figure out what part of Graylog is trying to call /api/system/metrics/multiple on the Master that causes these WARN messages, since nothing appears to be failing or be inaccessible from the Graylog Web Interface…