ProxiedResource - System information is currently unavailable

[Oirbsiu]

I have my Graylog install running on https with a valid Digicert RapidSSL and as far as I can tell, its all working. My inputs are collecting, my Sidecars are harvertings/collecting yet the only error that I get is when I view system / nodes. my logs continually show

2021-09-28T00:42:02.199+01:00 WARN  [ProxiedResource] Unable to call https://graylog.HOST:9000/api/system on node <ca5f4bcc-854f-42b3-92a5-8970fce503ab>: timeout
2021-09-28T00:42:12.271+01:00 WARN  [ProxiedResource] Unable to call https://graylog.HOST:9000/api/system on node <ca5f4bcc-854f-42b3-92a5-8970fce503ab>: timeout
2021-09-28T00:42:22.209+01:00 WARN  [ProxiedResource] Unable to call https://graylog.HOST:9000/api/system/metrics/multiple on node <ca5f4bcc-854f-42b3-92a5-8970fce503ab>: timeout
2021-09-28T00:42:32.271+01:00 WARN  [ProxiedResource] Unable to call https://graylog.HOST:9000/api/system/metrics/multiple on node <ca5f4bcc-854f-42b3-92a5-8970fce503ab>: timeout

I believe that I have all my configuration correct
/etc/sysconfig/graylog-server

# Default Java options for heap and garbage collection.
GRAYLOG_SERVER_JAVA_OPTS="-Xms4g -Xmx4g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/graylog/server/certificates/cacerts.jks -Djavax.net.ssl.trustStorePassword=changeit"

/etc/graylog/server/server.conf

http_bind_address = 192.168.16.22:9000
http_publish_uri = https://graylog.HOST:9000/
http_external_uri = https://graylog.HOST:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/graylog/server/certificates/cert.pem
http_tls_key_file = /etc/graylog/server/certificates/key.pem
elasticsearch_hosts = http://192.168.16.22:9200

other that the errors in the logs above, I do not see any other pointers that explain why I cannot access the system information.

any suggestion on what I should look for?

[gsmith]

Hello,

Maybe a couple things to try for troubleshooting.

The first thing I always check is permissions.

• Make sure graylog has access to its root directory.

chown graylog:graylog -R /etc/graylog/

• Make sure Graylog has access to the keystore.

• Graylog Server configuration file, did you set tls key password? (don’t forget to restart service)

http_tls_key_password = changeit

• Are you able to reach Graylog node itself?

curl -i https://GraylogIP:9000/api/system

If none of the above help, maybe try the following to see if that works.

• Do you get the same results using the IP address in you URL?

https://192.168.16.22:9000

• Have you tried the following

http_bind_address = 0.0.0.0:9000

Hope that helps.

[Oirbsiu]

@gsmith

curl -i https://GraylogIP:9000/api/system

Thanks for the reply.
My gut feeling is related to DNS lookup - the server has no external access currently and there are no local DNS servers on the same segment.

I have the FDQN name added to my /etc/hosts and for all the local tests (ping ssh etc…) I can reach the host using the FDQN. the only failure is with NSLOOKUP / dig (as I have no name servers)

Ive tried adding - thinking this may work (Removed APIs, Features, and Options)

 -Djdk.net.hosts.file=/etc/hosts

to my /etc/sysconf/graylog-server file but looks to have made no difference.

Any other suggestions?

answers to checks…

curl -i https://192.168.16.22:9000/api/system
curl: (51) SSL: no alternative certificate subject name matches target host name '192.168.16.22'

curl -i https://graylog.HOST:9000/api/system
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm="Graylog Server"
X-Graylog-Node-ID: ca5f4bcc-854f-42b3-92a5-8970fce503ab
X-Runtime-Microseconds: 11291
Content-Length: 0

curl -u admin -i https://graylog.HOST:9000/api/system
Enter host password for user 'admin':
HTTP/1.1 200 OK
X-Graylog-Node-ID: ca5f4bcc-854f-42b3-92a5-8970fce503ab
X-Runtime-Microseconds: 20152424
Content-Type: application/json
Content-Length: 388

{
	"facility": "graylog-server",
	"codename": "Noir",
	"node_id": "ca5f4bcc-854f-42b3-92a5-8970fce503ab",
	"cluster_id": "53a620e8-d2bb-4b30-8526-6400d04b382c",
	"version": "4.1.0+4eb2147",
	"started_at": "2021-09-28T11:26:49.400Z",
	"hostname": "graylog.HOST",
	"lifecycle": "running",
	"lb_status": "alive",
	"timezone": "Europe/Isle_of_Man",
	"operating_system": "Linux 4.18.0-310.el8.x86_64",
	"is_processing": true
}

I also tried commenting out “http_external_uri” e.g.

# Default: $http_publish_uri
#http_external_uri = https://graylog.HOST:9000/

but this resulted in these errors in the log - further demonstrating an issues with DNS

2021-09-28T16:25:15.286+01:00 WARN  [ProxiedResource] Unable to call https://192.168.16.22:9000/api/system/metrics/multiple on node <ca5f4bcc-854f-42b3-92a5-8970fce503ab>: Hostname 192.168.16.22 not verified:
    certificate: sha256/qjVKSlt/FfuNLEQ/9CjhcE5GyLEMa1whEBbtal3kkIE=
    DN: CN=*.HOST
    subjectAltNames: [*.HOST, HOST]

I did however have a button for the API browser…

[gsmith]

Hello,

@Oirbsiu

My gut feeling is related to DNS lookup

I might have to agree with you, hence why I was asking those questions I was trying to narrow it down.
Fortunately I’m spoiled at my work and I have a large amount of resources to my needs. AD, DNS, etc… within my lab. The best I can do is show you my Lab-Graylog setup.

graylog_config

http_bind_address = graylog.domain.com:9000
http_publish_uri = https://graylog.domain:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/pki/tls/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/pki/tls/certs/graylog/graylog-key.pem
http_tls_key_password = secret

Hosts_file

[root@graylog graylog_user]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
20.20.20.20 graylog.domain.com

hostname

[root@graylog graylog_user]# hostname
graylog.domain.com
[root@graylog graylog_user]#

I had to dig through my documents on this and it stated “Make sure I have Reverse DNS lookup in my DNS server”
Just to make sure did you double check you keystore to making sure Graylog has access?

root# ls -l /etc/graylog/server/certificates/cacerts.jks

I dug up my old post from version 2.0. Even thou the server.conf file has changed maybe there might be something in there that may help you. But I believe the error is the same.

Graylog2 Using https

Hello All, I am having a problem with https on graylog2.2.3, Currently using CentOS-7.3 all in one graylog2 server. This is a Virtual machine with 8 cores, 10 GB of ram and 1 Tb storage. I created an Input for Localhost and tested to received messages, completed. No problems. I attempted to use the following: http://docs.graylog.org/en/2.2/pages/configuration/https.html Unfortunately, toward the end of the page I was having problems with JVM to pick up the new trust store. So, I attempted t…

Hope that helps

EDIT: I forgot to ask you, did you check you certs in the keystore?

keytool -list -v -keystore keystore.jks -alias graylog.example.com

Here is an example of my old one. Notice my SubjectAlternativeName.

Cert_info

[root@graylog graylog]#  keytool -list -v -keystore graylog_keystore.jks  -alias graylog.domain.com
Enter keystore password: secret
Alias name: graylog.domain.com
Creation date: Sep 24, 2021
Entry type: trustedCertEntry

Owner: CN=graylog.domain.com, OU=admin, O=labs, L=cedar rapids, ST=iowa, C=us
Issuer: CN=graylog.domain.com, OU=admin, O=labs, L=cedar rapids, ST=iowa, C=us
Serial number: a7960790f745f617
Valid from: Wed Dec 16 16:52:41 CST 2020 until: Sat Dec 16 16:52:41 CST 2023
Certificate fingerprints:
         SHA1: 22:30:C4:D3:56:3C:7C:4D:7A:82:3F:36:AF:A2:6C:29:29:3C:C1:C5
         SHA256: FD:34:CC:DD:AC:7A:83:3D:11:70:FC:8C:E6:30:43:1B:4A:59:48:FC:B1:A1:0A:E8:0B:72:79:40:1E:CA:C5:91
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 0A C6 A4 E7 29 38 B7 1A   67 13 33 5D 9E 61 81 2B  ....)8..g.3].a.+
0010: 42 8E B5 D2                                        B...
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

#3: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: graylog.domain.com
  IPAddress: 20.20.20.20
]

#4: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 0A C6 A4 E7 29 38 B7 1A   67 13 33 5D 9E 61 81 2B  ....)8..g.3].a.+
0010: 42 8E B5 D2                                        B...
]
]

[Oirbsiu]

Definitely DNS
I managed to get ports opened on our firewall to access our local DNS server.
Added the record to the zone and almost straight away, the system / node information is accessible.

Perhaps this is a feature that could be looked into for a future release?

Happy to close this - Thanks @gsmith for your suggestions.

[gsmith]

Awesome, Glad I can help.