Inputs not working after using reverse proxy

Graylog Central (peer support)
1

Hi,

I finally got dockerized Graylog working behind proxy. However after doing that, GELF UDP input isn’t working anymore. Previously Graylog was using direct port and everything was working as expected.

I see the following error in the logs:

2018-03-02 14:55:45,252 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://172.22.0.4:9000/graylog/api/system/inputstates on node <3238f6f1-8179-45e5-820d-3d0b54805466>
java.net.SocketTimeoutException: timeout
at okio.Okio$4.newTimeoutException(Okio.java:230) ~[graylog.jar:?]
at okio.AsyncTimeout.exit(AsyncTimeout.java:285) ~[graylog.jar:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:241) ~[graylog.jar:?]
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345) ~[graylog.jar:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217) ~[graylog.jar:?]
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211) ~[graylog.jar:?]
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) ~[graylog.jar:?]
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) ~[graylog.jar:?]
at okhttp3.RealCall.execute(RealCall.java:69) ~[graylog.jar:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:204) ~[?:1.8.0_141]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_141]
at okio.Okio$2.read(Okio.java:139) ~[graylog.jar:?]
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[graylog.jar:?]

Compose File:

graylog:
    image: graylog:latest
    container_name: graylog
    environment:
        - GRAYLOG_REST_LISTEN_URI=http://0.0.0.0:9000/graylog/api/
        - GRAYLOG_WEB_LISTEN_URI=http://0.0.0.0:9000/graylog/
        - GRAYLOG_WEB_ENDPOINT_URI=http://site.com/graylog/api/
        - GRAYLOG_ELASTICSEARCH_HOSTS=http://elasticsearch:9200
        - GRAYLOG_ROOT_TIMEZONE=US/Eastern
        - "DISABLE_TELEMETRY: '1'"
        - GRAYLOG_ROTATION_STRATEGY=time
        - GRAYLOG_ELASTICSEARCH_MAX_TIME_PER_INDEX=1d
        - GRAYLOG_MAX_NUMBER_OF_INDICES=20
    links:
        - mongodb:mongo
        - elasticsearch
    depends_on:
        - mongodb
        - elasticsearch
    expose:
        - 9000
    ports:
        - 9000:9000
        - 1513:1513
        - 1513:1513/udp

Been trying to make Graylog work with Proxy for a while and hoping someone can guide on fixing this last issue.

2

the error is not an indicator that the input is not “working” - With the given information it is even hard to help you.

My Lab is running based on docker with the following docker-compose maybe this will help you to identify your problems.

regards
Jan

3

Thanks a lot Jan,

You are right, looks like messages started coming in after a while…which is weird. Not sure if now I am using a proxy that there is a big delay. If I use a direct port, the messages come in instantly.

Do you know if these two errors are concerning?

2018-03-02 14:55:45,252 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://172.22.0.4:9000/graylog/api/system/inputstates on node <3238f6f1-8179-45e5-820d-3d0b54805466
java.net.SocketTimeoutException: timeout

2018-03-02 15:53:27,209 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://172.22.0.4:9000/graylog/api/system/jobs on node <3238f6f1-8179-45e5-820d-3d0b54805466>
java.net.SocketTimeoutException: timeout

Thank

4

Try specifying the rest_transport_uri if http://172.22.0.4:9000/graylog/api/ isn’t the correct URI of the Graylog REST API which is reachable by all of your Graylog nodes.

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.