When I go to the page System / Nodes it says “System information is currently unavailable” below my single node.
The graylog server logs says:
WARN [ProxiedResource] Unable to call http://ip:9000/api/system on node : timeout
I’ve checked with the developper console on my browser and the timeout is exactly 10 seconds.
Then many pages have issues, I need to restart the graylog-server service to get Graylog fine again (except the Nodes page of course).
There is no reverse proxy and no firewall between my browser and Graylog, it’s a direct access.
Strange thing: when I try a simple curl on this URI directly executed on the Graylog server I get a good answer (code 200 and a response body) but after a very very long time: 4 minutes!!!
My environment:
CentOS 7.9
Java OpenJDK 1.8.0_312
Elasticsearch OSS 7.10
MongoDB 4.4
Graylog 4.2.5
Graylog plugins: graylog-plugin-collector-4.2.5 and graylog-storage-elasticsearch7-4.2.5
It’s a fresh install (it isn’t an upgrade).
I’ve found multiple topics on this subject, and there can be multiples causes, but I checked all of them and I can’t resolve my issue.
It’s not a DNS issue as I access Graylog directly with its IP address.
I set http_bind_address = ip:9000
I didn’t set http_publish_uri nor http_external_uri as by default they should use http_bind_address.
(I tried to set both of them but the result is the same)
It’s not a resource issue because I don’t receive any log.
The host has 8 CPU and 16 GB RAM.
Graylog heap size = 2 GB (I tried 3 GB but same issue)
Elastic heap size = 3 GB
Load average = 0.04
4 GB free RAM.
It’s not a certificate issue as I don’t use https.
Both services are running:
(graylog service was started few minutes ago because I restarted it after some tests like after chown but the issue is still there)
● graylog-server.service - Graylog server
Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2022-02-02 10:02:55 CET; 7min ago
Docs: http://docs.graylog.org/
Main PID: 36718 (graylog-server)
Tasks: 180 (limit: 49452)
Memory: 1.5G
CGroup: /system.slice/graylog-server.service
├─36718 /bin/sh /usr/share/graylog-server/bin/graylog-server
└─36753 /usr/bin/java -Xms2g -Xmx2g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -jar -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=rpm /usr/share/graylog-server/graylog.jar server -f /etc/graylog/server/server.conf -np
Feb 02 10:02:55 graylog systemd[1]: Started Graylog server.
About the /etc/hosts file are you talking about the one on the Graylog server or the one on my workstation ?
I don’t understand why I should add it because I access graylog directly to its IP address (http://192.168.0.2:9000) and this IP is set for the http_bind_adress.
I don’t find any error in Elasticsearch logs (/var/log/elasticsearch/graylog.log and journalctl -u elasticsearch).
Unfortunately it doesn’t help because those issues where related to bad use of rest_xxx and web_xxx which were the old configuration syntax (it has been replaced by http_xxx which is easier).
If you can login the Web UI and Graylog is unable to Call that URI(http://ip:9000/api/system). then it might be API problem with that URL. This could be a couple different issues. Either its being block or Graylog is unable to find that Address. Configuring your /etc/hosts might help. Whenever you open a website by typing its hostname/ipaddress, your system will read through the hosts file to check for the corresponding IP and then open it. If your hostname is called localhost some thing like this should be configured.
192.168.0.2 localhost
OR
192.168.0.2 myhostname
Restart network service or reboot.
I personally use IP address and make my network static. This prevents issue in the future.
I’m using GL 4.2, ES 7.10, Mongo 4.4 here is my environment.
Perhaps check all directory’s that Graylog would need to use, specially plugins, ect… for permission issues.
Check system journal by executing this journalctl -xe looking for anything that would pertain to connecting IP ADD 192.168.0.2 or something similar.
Restart services and tail the log files elasticsearch, Graylog and MongoDb etc…
Next, I know its default but I would try to set this in my Graylog configuration file http_enable_cors = true and restart GL service.
I’m assuming your using localhost instead of 127.0.0.1 for Graylog and Elasticsearch connection? If so, then I would make sure the my etc/hosts file know what localhost is
Example
Yes I can login the Web UI and I don’t have any problem with others pages. The only issue is with this API call.
I’ve checked if hotsname were used somewhere to replace them with IP addresses to not have any problem with hostname resolution.
First I’ve found mongodb_uri = mongodb://localhost/graylog.
So I set mongodb_uri = mongodb://127.0.0.1/graylog.
(It may be not part of the solution because in /etc/hosts there is by default 127.0.0.1 localhost)
Finally I restarted Graylog to apply this 2 settings.
And it works!
But it’s strange, if I revert those settings it’s still work…
Maybe it’s also stored somewhere in MongoDB (I didn’ find where).
In my 2nd Graylog I use TLS and I’ve found a typo in javax.net.ssl.trustStore (I typed “ss” instead of “ssl”…).
It’s really hard to find the root cause of this issue.
It can be many things, and logs don’t help, they only say “timeout”.
And I don’t understand why this specific page acts diffrently than other pages.
