Scenario:
- OS Information: Ubuntu 22.04.2 LTS fresh installation
- docker v20.10.21
- docker-compose v1.29.2
- new GrayLog installation using docker compose
Issue:
the graylog server is reachable via http://[internal_ip_address]:9090 but nothing than a white page
Troubleshootings tried on graylog server:
- Result of “sudo netstat -lntp”:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:12201 0.0.0.0:* LISTEN 157153/docker-proxy
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 4871/systemd-resolv
tcp 0 0 127.0.0.1:34599 0.0.0.0:* LISTEN 17224/containerd
tcp 0 0 0.0.0.0:1514 0.0.0.0:* LISTEN 157234/docker-proxy
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 16538/sshd: /usr/sb
tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN 157212/docker-proxy
tcp6 0 0 :::12201 :::* LISTEN 157161/docker-proxy
tcp6 0 0 :::1514 :::* LISTEN 157240/docker-proxy
tcp6 0 0 :::22 :::* LISTEN 16538/sshd: /usr/sb
tcp6 0 0 :::9000 :::* LISTEN 157218/docker-proxy
- Result of “curl -i http://127.0.0.1:9000”:
HTTP/1.1 200 OK
X-UA-Compatible: IE=edge
X-Graylog-Node-ID: c386dafe-88b6-4312-aac1-37a2f0043fe0
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-eval'; img-src data: *; connect-src *
Content-Type: text/html
Content-Length: 1593
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="robots" content="noindex, nofollow">
<meta charset="UTF-8">
<title>Graylog Web Interface</title>
<link rel="shortcut icon" href="/assets/favicon.png">
</head>
<body>
<div id="app-root" />
<script src="/config.js"></script>
<script src="/assets/vendor.8a712d83b311029bb0b6.js"></script>
<script src="/assets/polyfill.b953b8f8d9507ea6486e.js"></script>
<script src="/assets/plugin/org.graylog.plugins.collector.CollectorPlugin/plugin.org.graylog.plugins.collector.CollectorPlugin.3d73be4ad22bcd245615.js"></script>
<script src="/assets/plugin/org.graylog.plugins.collector.CollectorPlugin/d7f3395e-765.8a1c541b3d33ef082e9b.js"></script>
<script src="/assets/plugin/org.graylog.integrations.IntegrationsPlugin/plugin.org.graylog.integrations.IntegrationsPlugin.068db92b12f40069e5f6.js"></script>
<script src="/assets/plugin/org.graylog.integrations.IntegrationsPlugin/06377e61-300.177f754b22f4c5c567e3.js"></script>
<script src="/assets/plugin/org.graylog.aws.AWSPlugin/plugin.org.graylog.aws.AWSPlugin.b729e45314bd65faaa0f.js"></script>
<script src="/assets/plugin/org.graylog.aws.AWSPlugin/a09fd9c1-300.e1674da2e4f972ee06e7.js"></script>
<script src="/assets/app.ec2c9fc60785a65a39e6.js"></script>
<script src="/assets/804bf16d-924.f70ecb144a779af893b6.js"></script>
<script src="/assets/804bf16d-4012.9923a21a06a6f13457d9.js"></script>
</body>
</html>
- Same result executing "curl -i http://[internal_ip_address]:9000
If I call http://[internal_ip_address]:9000 from an allowed client I can see this error trace:
graylog_1 | 2023-07-11 12:36:56,644 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
graylog_1 | 2023-07-11 12:42:18,137 ERROR: org.glassfish.jersey.server.ServerRuntime$Responder - An I/O error has occurred while writing a response message entity to the container output stream.
graylog_1 | org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection closed
graylog_1 | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:67) ~[graylog.jar:?]
graylog_1 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139) ~[graylog.jar:?]
graylog_1 | at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116) ~[graylog.jar:?]
graylog_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:642) [graylog.jar:?]
graylog_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:373) [graylog.jar:?]
graylog_1 | at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:363) [graylog.jar:?]
graylog_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:258) [graylog.jar:?]
graylog_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
graylog_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
graylog_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
graylog_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
graylog_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
graylog_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?]
graylog_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [graylog.jar:?]
graylog_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [graylog.jar:?]
graylog_1 | at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?]
graylog_1 | at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
graylog_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
graylog_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
graylog_1 | at java.lang.Thread.run(Unknown Source) [?:?]
graylog_1 | Caused by: java.io.IOException: Connection closed
graylog_1 | at org.glassfish.grizzly.asyncqueue.TaskQueue.onClose(TaskQueue.java:307) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.onClose(AbstractNIOAsyncQueueWriter.java:477) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.transport.TCPNIOTransport.closeConnection(TCPNIOTransport.java:379) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.NIOConnection.doClose(NIOConnection.java:643) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.NIOConnection$6.run(NIOConnection.java:609) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.DefaultSelectorHandler.execute(DefaultSelectorHandler.java:213) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.NIOConnection.terminate0(NIOConnection.java:603) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.transport.TCPNIOConnection.terminate0(TCPNIOConnection.java:267) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:112) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:82) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:320) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:84) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:515) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:79) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:66) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:391) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:360) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:324) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:255) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) ~[graylog.jar:?]
graylog_1 | ... 1 more
graylog_1 | Caused by: java.io.IOException: Connection reset by peer
graylog_1 | at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
graylog_1 | at sun.nio.ch.SocketDispatcher.write(Unknown Source) ~[?:?]
graylog_1 | at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source) ~[?:?]
graylog_1 | at sun.nio.ch.IOUtil.write(Unknown Source) ~[?:?]
graylog_1 | at sun.nio.ch.IOUtil.write(Unknown Source) ~[?:?]
graylog_1 | at sun.nio.ch.SocketChannelImpl.write(Unknown Source) ~[?:?]
graylog_1 | at org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:125) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.transport.TCPNIOUtils.writeCompositeBuffer(TCPNIOUtils.java:64) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:105) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:82) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:320) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:84) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:515) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:79) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:66) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:391) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:360) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:324) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:255) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) ~[graylog.jar:?]
graylog_1 | at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) ~[graylog.jar:?]
Below my docker-compose.yml:
version: '2'
services:
# MongoDB: https://hub.docker.com/_/mongo/
mongodb:
image: mongo:5.0.13
volumes:
- mongo_data:/data/db
# Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/docker.html
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
volumes:
- es_data:/usr/share/elasticsearch/data
environment:
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
mem_limit: 1g
# Graylog: https://hub.docker.com/r/graylog/graylog/
graylog:
image: graylog/graylog:5.1
volumes:
- graylog_data:/usr/share/graylog/data
environment:
- GRAYLOG_PASSWORD_SECRET=[xxxxxxxxxxxxxxxxxxxxxxxxxxx]
- GRAYLOG_ROOT_PASSWORD_SHA2=[xxxxxxxxxxxxxxxxxxxxxxxxxxx]
- GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
- GRAYLOG_HTTP_BIND_ADDRESS=0.0.0.0:9000
- GRAYLOG_ELASTICSEARCH_HOSTS=http://elasticsearch:9200
- GRAYLOG_MONGODB_URI=mongodb://mongodb:27017/graylog
#entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 -- /docker-entrypoint.sh
entrypoint: /docker-entrypoint.sh
links:
- mongodb:mongo
- elasticsearch
restart: always
depends_on:
- mongodb
- elasticsearch
ports:
- 9000:9000 # Graylog web interface and REST API
- 1514:1514 # Syslog TCP
- 1514:1514/udp # Syslog UDP
- 12201:12201 # GELF TCP
- 12201:12201/udp # GELF UDP
# Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
mongo_data:
driver: local
es_data:
driver: local
graylog_data:
driver: local
Side note: I had to comment out the “entrypoint” line because otherwise that line returned an error: the docker-entrypoint.sh file is missing
Any help is really appreciated!
Thank you