WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://192.168.0.165:9000/api/system/metrics/multiple on node

Graylog Central (peer support)
1

Hi,
I have graylog-3.3.8, elesticsearch-oss-6.8.10 and mongo-4 containers loaded from a docker-compose file (using podman-compose on CentOS 8).

I have managed to get https working with a self signed certificate.

The logs are fine until I actually log into the web gui, then I recieve repeated messaged every 2 secs like

2020-11-04 10:04:44,851 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://192.168.0.165:9000/api/system/metrics/multiple on node : sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.

This following are the out from curl when trying to access api. /api, /api/system, /api/system/metrics are all accessible with username and password, however /api/system/metrics/multiple returns Method Not Allowed. What is trying to access /api/system/metrics/multiple ?

curl -ik https://192.168.0.165:9000
HTTP/1.1 200 OK
X-UA-Compatible: IE=edge
X-Graylog-Node-ID: e1b76d8f-4e3e-41d9-96c8-20596e613504
Content-Type: text/html
Date: Wed, 04 Nov 2020 10:07:22 GMT
Content-Length: 1359

Graylog Web Interface 
<script src="https://192.168.0.165:9000/assets/vendor.8d6aa5835c8302c41e2f.js"></script>

<script src="https://192.168.0.165:9000/assets/polyfill.4678e54c30f1075e4dd8.js"></script>

<script src="https://192.168.0.165:9000/assets/builtins.4678e54c30f1075e4dd8.js"></script>

<script src="https://192.168.0.165:9000/assets/plugin/org.graylog.plugins.threatintel.ThreatIntelPlugin/plugin.org.graylog.plugins.threatintel.ThreatIntelPlugin.3c268beb1370706a386c.js"></script>

<script src="https://192.168.0.165:9000/assets/plugin/org.graylog.plugins.collector.CollectorPlugin/plugin.org.graylog.plugins.collector.CollectorPlugin.15b0bd7e31cbfef24c6f.js"></script>

<script src="https://192.168.0.165:9000/assets/plugin/org.graylog.aws.AWSPlugin/plugin.org.graylog.aws.AWSPlugin.f2cd3046f5d8c08ce593.js"></script>

<script src="https://192.168.0.165:9000/assets/app.4678e54c30f1075e4dd8.js"></script>

curl -ik https://192.168.0.165:9000/api/system -u admin:Secret10!1220MustChange8989
HTTP/1.1 200 OK
X-Graylog-Node-ID: e1b76d8f-4e3e-41d9-96c8-20596e613504
X-Runtime-Microseconds: 3921
Content-Type: application/json
Date: Wed, 04 Nov 2020 10:11:52 GMT
Content-Length: 383

{“facility”:“graylog-server”,“codename”:“Sloth Rocket”,“node_id”:“e1b76d8f-4e3e-41d9-96c8-20596e613504”,“cluster_id”:“eb2e01cf-e71c-4a7b-b17b-74680dc5b22a”,“version”:“3.3.8+e223f85”,“started_at”:“2020-11-04T09:32:09.962Z”,“hostname”:“Unknown”,“lifecycle”:“running”,“lb_status”:“alive”,“timezone”:“Etc/UTC”,“operating_system”:“Linux 4.18.0-193.14.2.el8_2.x86_64”,“is_processing”:true}

curl -ik https://192.168.0.165:9000/api/system/metrics/multiple -u admin:Secret10!1220MustChange8989
HTTP/1.1 405 Method Not Allowed
Allow: POST,OPTIONS
X-Graylog-Node-ID: e1b76d8f-4e3e-41d9-96c8-20596e613504
Content-Type: application/json
Date: Wed, 04 Nov 2020 10:13:09 GMT
Content-Length: 59

{“type”:“ApiError”,“message”:“HTTP 405 Method Not Allowed”}

As well as GRAYLOG_PASSWORD_SECRET and GRAYLOG_ROOT_PASSWORD_SHA2, the docker-compose also sets
- GRAYLOG_HTTP_EXTERNAL_URI=https://192.168.0.165:9000/
- GRAYLOG_HTTP_PUBLISH_URI=https://192.168.0.165:9000/
- GRAYLOG_HTTP_ENABLE_TLS=true
- GRAYLOG_HTTP_TLS_CERT_FILE=/usr/share/graylog/ssl/certs/cert.pem
- GRAYLOG_HTTP_TLS_KEY_FILE=/usr/share/graylog/ssl/private/pkcs8-key.pem

Are there other variable that needs to be set when using https?

2

He @mnsmithuk

the root of your problem is here:

as you have a self signed certificate - your Graylog is not able to verify that certificate when it connects to itself.

https://docs.graylog.org/en/4.0/pages/configuration/https.html#adding-a-self-signed-certificate-to-the-jvm-trust-store

the above is the docs that will explain that.

3

Jan,

I had already followed the documentation but it still was not working.

For anyone who has been having this problem, they way I fixed it was to change my initial environment variable

GRAYLOG_SERVER_JAVA_OPTS="-Djavax.net.ssl.trustStore=/usr/share/graylog/ssl/certs/graylog-cacerts.jks -Djavax.net.ssl.trustStorePassword=secret"

to

“GRAYLOG_SERVER_JAVA_OPTS=-Djavax.net.ssl.trustStore=/usr/share/graylog/ssl/certs/graylog-cacerts.jks -Djavax.net.ssl.trustStorePassword=secret”

in my docker-compose file. It’s just a subtle difference in the position of the " in the docker-compose.yml but made the difference.

It seems as if all the / get subsituted with a . when the quote comes after the = sign and therefore graylog could not find the ssl trustStore cert and secret. Not sure if that is a bug in docker-entrypoint.sh, docker-compose.yml or podman-compose/docker-compose but moving the " to the beginning fixed al my https issues.

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.