Graylog 3.0 Docker container behind Nginx reverse proxy

lmm5247 · April 2, 2019, 7:04pm

I just setup Graylog 3.0 in a Docker container and the server seems to start without issue (below is my docker-compose.yml file). Nginx, Mongo, and Elasticsearch are all on separate machines.

version: '3'
services:
  graylog:
    container_name: graylog
    restart: unless-stopped
    networks:
      - home
    ports:
      - '9000:9000'
      - '12201:12201'
      - '12201:12201/udp'
      - '1514:1514'
      - '1514:1514/udp'
      - '8514:8514/udp'
    environment:
      - 'GRAYLOG_MONGODB_URI=mongodb://mongodb:27017/graylog'
      - 'GRAYLOG_ELASTICSEARCH_HOSTS=http://elasticsearch:9200'
      - 'GRAYLOG_PASSWORD_SECRET=97B4F23C7FCA34776EA399DAF5D3B384F4C64B3EA723DD28AAFE5F2ECB559310'
      - 'GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918'
      - 'GRAYLOG_HTTP_BIND_ADDRESS=0.0.0.0:9000'
      - 'GRAYLOG_HTTP_PUBLISH_URI=http://10.10.2.21:9000/'
      - 'GRAYLOG_HTTP_EXTERNAL_URI=https://graylog.internal.domain.com/'
    volumes:
      - graylog_data:/usr/share/graylog/data
    image: graylog/graylog:3.0

networks:
  home:
    external: true

volumes:
  graylog_data:
    driver: local

I can login at https://graylog.internal.domain.com/ via my Nginx reverse proxy. This Nginx instance is not on the same container as Greylog, it is on another machine. The address 10.20.2.21 is my Docker host.

  location / {
    proxy_pass         http://10.10.2.21:9000/;
    proxy_set_header   Host                  $host;
    proxy_set_header   X-Forwarded-For       $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Host      $host;
    proxy_set_header   X-Forwarded-Server    $host;
  }
}

When I try to access anything with the API though, I see errors in the logs that look like this. What am I missing here? As a mentioned, the web interface works over HTTPS, but not when it needs to reference any data from the API.

2019-04-02 18:55:31,434 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://10.10.2.21:9000/api/system on node <8a0e6bb2-a894-4acd-93ab-65eab8d24e65>,
java.net.SocketTimeoutException: connect timed out,
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_212],
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_212],
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_212],
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_212],
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_212],
	at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_212],
	at okhttp3.internal.platform.Platform.connectSocket(Platform.java:129) ~[graylog.jar:?],
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:245) ~[graylog.jar:?],
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:165) ~[graylog.jar:?],
	at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) ~[graylog.jar:?],
	at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) ~[graylog.jar:?],
	at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) ~[graylog.jar:?],
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ~[graylog.jar:?],
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?],
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?],
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?],
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?],
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?],
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?],
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?],
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) ~[graylog.jar:?],
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?],
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?],
	at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:61) ~[graylog.jar:?],
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) ~[graylog.jar:?],
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) ~[graylog.jar:?],
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) ~[graylog.jar:?],
	at okhttp3.RealCall.execute(RealCall.java:77) ~[graylog.jar:?],
	at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?],
	at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:78) ~[graylog.jar:?],
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212],
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212],
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212],
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212],

benvanstaveren · April 3, 2019, 10:59am

You’ll want to add this:

proxy_set_header X-Graylog-Server-URL https://$server_name;

lmm5247 · April 3, 2019, 12:43pm

@benvanstaveren - I added that header, but I’m still getting the same errors in the container log.

2019-04-03 12:42:50,374 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://10.10.2.21:9000/api/system/metrics/multiple on node <8a0e6bb2-a894-4acd-93ab-65eab8d24e65>

java.net.SocketTimeoutException: connect timed out

benvanstaveren · April 4, 2019, 6:51am

Hmm, are you sure your settings regarding listen url etc. etc. are correct? Throw your server.conf on pastebin (remove all passwords etc. before you do) and drop the link here, it’ll give me some more info to work with

jan · April 4, 2019, 8:28am

is your Graylog able to connect to itself via:

GRAYLOG_HTTP_PUBLISH_URI=http://10.10.2.21:9000/

Because you configured that Graylog can reach itself on that IP/PORT … please read the comments for this configuration settings.

lmm5247 · April 4, 2019, 12:17pm

@jan and @benvanstaveren - Thanks for the replies, but you can close this. I installed Graylog on a VM and have no issues with the same settings, so it must be something with my Docker setup…

system · April 18, 2019, 12:19pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog docker container behind an Nginx reverse proxy in a sub-directory Graylog Central (peer support)	6	5460	June 1, 2018
Graylog container behind Nginx reverse-proxy container Graylog Central (peer support)	2	993	December 20, 2020
Graylog 3.0 with docker container Graylog Central (peer support) sidecar , nxlog , filebeat-linux , filebeat-windows , winlogbeat , nodatanx	9	5362	March 21, 2019
Graylog 3.0.2 inside docker with different port Graylog Central (peer support)	4	1432	July 31, 2019
Graylog Docker NGINX Reverse Proxy HTTPS Graylog Central (peer support)	2	2827	February 14, 2020

Graylog 3.0 Docker container behind Nginx reverse proxy

Related topics