Unable to call http://<domain>:9000/api/system

Graylog Central (peer support)
1

Hello,

I am experiencing a problem where the majority of the services are running fine, but the system (and cluster) api endpoint does not work as intended.

I don’t see System information in the Nodes view on the GUI
The In / Out msg/s is not working properly

image

This is my server.config endpoint setup:

rest_listen_uri = http://IP:12900/
rest_transport_uri is disabled
web_listen_uri = http://IP:9000
web_endpoint_uri = htt p://IP:12900/

(I have tried several combinations of uri)

And last but not least my server.log

2017-11-28T15:15:07.047+01:00 WARN  [ProxiedResource] Unable to call http://IP:12900/system on node <88fcfc8b-e5fa-4494-8bd7-fdc57c09180b>
java.net.SocketTimeoutException: timeout
        at okio.Okio$4.newTimeoutException(Okio.java:230) ~[graylog.jar:?]
        at okio.AsyncTimeout.exit(AsyncTimeout.java:285) ~[graylog.jar:?]
        at okio.AsyncTimeout$2.read(AsyncTimeout.java:241) ~[graylog.jar:?]
        at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345) ~[graylog.jar:?]
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217) ~[graylog.jar:?]
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211) ~[graylog.jar:?]
        at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) ~[graylog.jar:?]
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
        at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) ~[graylog.jar:?]
        at okhttp3.RealCall.execute(RealCall.java:69) ~[graylog.jar:?]
        at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
        at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
Caused by: java.net.SocketException: Socket closed
        at java.net.SocketInputStream.read(SocketInputStream.java:203) ~[?:1.8.0_101]
        at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_101]
        at okio.Okio$2.read(Okio.java:139) ~[graylog.jar:?]
        at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[graylog.jar:?]
        ... 28 more

Specs:
SUSE Linux Enterprise Server 12 (x86_64)
Graylog: Graylog 2.3.2+3df951e
MongoDB: 3.2
Elasticsearch: 5.6.4#

Any help would be appreciated :slight_smile:

Edit:

The call curl -u admin:hunter2 http://IP:12900/api/system returns a HTML doc with some scripts in it (on the Greylog Server).

2

Which combinations have you tried?
What was the result?

Also, please elaborate on your network setup.

3

Initially, I had /api endpoint to the rest_listen_uri.
I tried setting the rest_transport_ui =http://IP:12900 which did not resolve the issue.
I also tried setting rest_listen_uri the same as web_listen_uri with reference http://docs.graylog.org/en/2.3/pages/configuration/web_interface.html#single-or-separate-listeners-for-web-interface-and-rest-api

I also disabled the web interface, but the api endpoints above were not accessible than either.

I have a single node Graylog installation. Currently, I am only sending Linux Syslog Messages from the same machine.
I entered the Graylog UI from another machine on the same subnet.

4

Update:
There is no firewall or proxy configured on/inside the net.

Since Graylog is struggling with fetching information about the same VM I checked the DNS.

nslookup GREYLOG_SERVER_IP
;; Got SERVFAIL reply from xxx.xxx.xxx.xxx, trying next server
;; Got SERVFAIL reply from xxx.xxx.xxx.xxx, trying next server
;; connection timed out; no servers could be reached

I tried hacking an entry in etc/hosts which resolves the server IP to graylog and then changed the uri in the web_endpoint_uri to http://graylog:12900
This caused the web ui to not be able to connect to the server anymore.

Any Guesses I can look after?
How does Graylog get the System Information required for the Nodes Detail View?
Any other logfiles I can loose myself in?

5

As long as you don’t provide details about the network setup or at least redact your configuration snippets and command outputs in a consistent way, there’s not much to help you.

6

Hello,

just coming back from vacation and the problem resolved itself somehow.

I kept this configuration over a week. When I came back from vacation, the Web UI was accessible and the system endpoint worked fine. I checked and there was no system reboot or anything.

The Topic can be closed. How do I close my topic?

7

Just don’t post any replies. Topics will be automatically closed 14 days after the last post.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.