Docker Graylog unhealthy

schneich · December 18, 2022, 12:33pm

Dear community,

I got to know Graylog at work. Recently, I was able to get my hands on an old HP Elitedesk 800 G2, which I want to use as a little server in our basement. I installed Ubuntu Server and Docker environment.
My first project ist “Graylog”. After 2 nights of try and error, I got it finaly to the login page of Graylog.

1. Describe your incident:
Unfortunatley, graylog is not reachable anymore. The container indicates a status of “unhealthy”. I am a newbee with Docker, so I am not very sure on how to debug this. Any ideas are very much welcome.

2. Describe your environment:

OS Information: Ubuntu 22.04.1 LTS
Service logs, configurations, and environment variables:
Here is my docker-compose.yml:

version: '2'
services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongodb:
    image: mongo:5.0.13
    networks:
      - graylog
  #DB in share for persistence
    volumes:
      - mongo_data:/data/db
   # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
    #data folder in share for persistence
    volumes:
      - es_data:/usr/share/elasticsearch/data
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
    networks:
      - graylog
  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:5.0
    #journal and config directories in local NFS share for persistence
    volumes:
      - graylog_journal:/usr/share/graylog/data/journal
    environment:
      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=[abcd]
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=[efgh]
      - GRAYLOG_HTTP_EXTERNAL_URI=http://192.168.1.57:9000/
      - GRAYLOG_HTTP_ENABLE_CORS=true
    entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh
    networks:
      - graylog
    links:
      - mongodb:mongo
      - elasticsearch
    restart: always
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 1514:1514
      # Syslog UDP
      - 1514:1514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp
# Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
  mongo_data:
    driver: local
  es_data:
    driver: local
  graylog_journal:
    driver: local
networks:
    graylog:
      driver: bridge

I took the basics from the docker-docs and enhanced it with some things I found on the internet. I have to admit that I do not understand all of it.

State of the container:

user@ubuntu:~$ docker ps
CONTAINER ID   IMAGE                 COMMAND                  CREATED        STATUS                      PORTS                                                                                                                                                                                                                           NAMES
694aa8348e37   graylog/graylog:5.0   "/usr/bin/tini -- wa…"   36 hours ago   Up 10 minutes (unhealthy)   0.0.0.0:1514->1514/tcp, :::1514->1514/tcp, 0.0.0.0:9000->9000/tcp, 0.0.0.0:1514->1514/udp, :::9000->9000/tcp, :::1514->1514/udp, 0.0.0.0:12201->12201/tcp, 0.0.0.0:12201->12201/udp, :::12201->12201/tcp, :::12201->12201/udp   graylog_graylog_1

3. What steps have you already taken to try and solve the problem?
Well, I deleted the container and re-created it with docker-compose up -d. If I remember right, the container took some minutes to start and after a while, I could access Graylog. After turning of the computer yesterday and starting it up today, the container shows unhealthy.

4. How can the community help?
Perhaps, someone can help with debugging this:

What is a normal startup-time, until a Docker Container is accessable?
What am I doing wrong? Why is my Graylog unhealthy?
Is there something wrong with my docker-compose.yml?

EDIT:
At the moment, the network is in “bridge”-mode. I do have a Unifi-network at home and I would like to test the macvlan-option. Is this relevant in debugging this? Should I first get a healthy Graylog and then start trying the macvlan-option or is this independent from each other?

schneich · December 18, 2022, 7:39pm

I just figured out, how to tail the logs for a container. There seems to be something with my mongo-db:

2022-12-18 19:33:29,866 INFO : org.graylog2.featureflag.ImmutableFeatureFlagsCollector - Following feature flags are used: {default properties file=[scripting_api_preview=off, search_filter=on]}
2022-12-18 19:33:30,336 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: AWS plugins 5.0.1 [org.graylog.aws.AWSPlugin]
2022-12-18 19:33:30,337 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Integrations 5.0.1 [org.graylog.integrations.IntegrationsPlugin]
2022-12-18 19:33:30,338 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Collector 5.0.1 [org.graylog.plugins.collector.CollectorPlugin]
2022-12-18 19:33:30,338 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Threat Intelligence Plugin 5.0.1 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2022-12-18 19:33:30,338 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elasticsearch 7 Support 5.0.1+f28b729 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2022-12-18 19:33:30,338 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: OpenSearch 2 Support 5.0.1+f28b729 [org.graylog.storage.opensearch2.OpenSearch2Plugin]
2022-12-18 19:33:30,357 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -Dlog4j2.formatMsgNoLookups=true -Djdk.tls.acknowledgeCloseNotify=true -XX:+UnlockExperimentalVMOptions -XX:-OmitStackTraceInFastThrow -XX:+UseG1GC -Dlog4j.configurationFile=/usr/share/graylog/data/config/log4j2.xml -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker
2022-12-18 19:33:30,492 INFO : org.mongodb.driver.client - MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.7.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.15.0-56-generic"}, "platform": "Java/Eclipse Adoptium/17.0.5+8"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@7a389761]}, clusterSettings={hosts=[mongo:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
2022-12-18 19:33:30,493 INFO : org.mongodb.driver.client - MongoClient with metadata {"driver": {"name": "mongo-java-driver|legacy", "version": "4.7.1"}, "os": {"type": "Linux", "name": "Linux", "architecture": "amd64", "version": "5.15.0-56-generic"}, "platform": "Java/Eclipse Adoptium/17.0.5+8"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=true, retryReads=true, readConcern=ReadConcern{level=null}, credential=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.Jep395RecordCodecProvider@7a389761]}, clusterSettings={hosts=[mongo:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='30000 ms', localThreshold='30000 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, sendBufferSize=0}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, sendBufferSize=0}, connectionPoolSettings=ConnectionPoolSettings{maxSize=1000, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=false, invalidHostNameAllowed=false, context=null}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, contextProvider=null}
2022-12-18 19:33:30,500 INFO : org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
2022-12-18 19:33:30,506 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongo:27017
com.mongodb.MongoSocketException: mongo
        at com.mongodb.ServerAddress.getSocketAddresses(ServerAddress.java:217) ~[graylog.jar:?]
        at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:75) ~[graylog.jar:?]
        at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[graylog.jar:?]
        at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:180) ~[graylog.jar:?]
        at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:193) [graylog.jar:?]
        at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:157) [graylog.jar:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.net.UnknownHostException: mongo
        at java.net.InetAddress$CachedAddresses.get(Unknown Source) ~[?:?]
        at java.net.InetAddress$NameServiceAddresses.get(Unknown Source) ~[?:?]
        at java.net.InetAddress.getAllByName0(Unknown Source) ~[?:?]
        at java.net.InetAddress.getAllByName(Unknown Source) ~[?:?]
        at java.net.InetAddress.getAllByName(Unknown Source) ~[?:?]
        at com.mongodb.ServerAddress.getSocketAddresses(ServerAddress.java:209) ~[graylog.jar:?]
        ... 6 more
2022-12-18 19:34:00,502 INFO : org.graylog2.bootstrap.preflight.MongoDBPreflightCheck - MongoDB is not available. Retry #1
2022-12-18 19:34:02,503 INFO : org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
2022-12-18 19:34:32,504 INFO : org.graylog2.bootstrap.preflight.MongoDBPreflightCheck - MongoDB is not available. Retry #2
2022-12-18 19:34:34,505 INFO : org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
2022-12-18 19:35:04,506 INFO : org.graylog2.bootstrap.preflight.MongoDBPreflightCheck - MongoDB is not available. Retry #3
2022-12-18 19:35:06,507 INFO : org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
2022-12-18 19:35:36,507 INFO : org.graylog2.bootstrap.preflight.MongoDBPreflightCheck - MongoDB is not available. Retry #4
2022-12-18 19:35:38,508 INFO : org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
2022-12-18 19:36:08,509 INFO : org.graylog2.bootstrap.preflight.MongoDBPreflightCheck - MongoDB is not available. Retry #5
2022-12-18 19:36:10,510 INFO : org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
2022-12-18 19:36:40,510 INFO : org.graylog2.bootstrap.preflight.MongoDBPreflightCheck - MongoDB is not available. Retry #6
2022-12-18 19:36:42,511 INFO : org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
2022-12-18 19:37:12,512 INFO : org.graylog2.bootstrap.preflight.MongoDBPreflightCheck - MongoDB is not available. Retry #7
2022-12-18 19:37:14,512 INFO : org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out

What does this mean?

StefanAustin · December 18, 2022, 7:59pm

Sadly, I never worked with Docker and Graylog. I know from my coworker he had issues with restarting the container.
Once he restarted his host, the container didn’t restart, so he had to do it manually.

Maybe you can try to restart mongo manually and then graylog after it?

schneich · December 18, 2022, 8:07pm

Hit @StefanAustin,

thank you very much for your reply. Well, I just hit docker-compose up -d et voilà, the login screen is available.

Am I right, that the graylog-container started with the boot of my ubuntu, but the mongo-db did not? How do I configre a auto-boot for mongo-DB and Elasticsearch?

Thanks

user@ubuntu:~/Docker/Graylog$ docker ps
CONTAINER ID   IMAGE                                                      COMMAND                  CREATED        STATUS                    PORTS                                                                                                                                                                                                                           NAMES
694aa8348e37   graylog/graylog:5.0                                        "/usr/bin/tini -- wa…"   44 hours ago   Up 31 minutes (healthy)   0.0.0.0:1514->1514/tcp, :::1514->1514/tcp, 0.0.0.0:9000->9000/tcp, 0.0.0.0:1514->1514/udp, :::9000->9000/tcp, :::1514->1514/udp, 0.0.0.0:12201->12201/tcp, 0.0.0.0:12201->12201/udp, :::12201->12201/tcp, :::12201->12201/udp   graylog_graylog_1
e4d249073031   mongo:5.0.13                                               "docker-entrypoint.s…"   44 hours ago   Up About a minute         27017/tcp                                                                                                                                                                                                                       graylog_mongodb_1
3e2daccf8dfe   docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2   "/tini -- /usr/local…"   44 hours ago   Up About a minute         9200/tcp, 9300/tcp                                                                                                                                                                                                              graylog_elasticsearch_1
user@ubuntu:~/Docker/Graylog$

StefanAustin · December 19, 2022, 3:04am

Sorry for answering late, but I do not know the answer.
Most likely it is just a single line in a config file, but I never used Docker besides in a home lab to test things.

lcosta · December 19, 2022, 9:32am

Hi there,

Can you check and show the values for:

docker ps

Check the id container for “mongodb” and show us the log

docker logs <container ID>

I haven´t still made the Graylog 5.0 work in container smooth.

Check this version of YML also (using opensearch):
https://github.com/Graylog2/docker-compose/blob/main/open-core/docker-compose.yml

gsmith · December 19, 2022, 11:38pm

Hey @schneich

you have something like this in compose file.


entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh
links:
 - mongodb:mongo
 - elasticsearch
 restart: always < ---- here
depends_on:
 - mongodb
 - elasticsearch

You can apply whatever Restart Policy you prefer after the –restart option, so:

–restart on-failure
–restart always
–restart unless-stopped

For mongoDb

 mongo1:
    hostname: mongo1
    container_name: mongo1
    image: 'mongodb/mongodb:latest'
    restart: unless-stopped (i.e., always)
    networks:

schneich · December 20, 2022, 8:12pm

Hi @lcosta and @gsmith,

thank you very much for your inputs.

@lcosta: I will try to follow the current path. If I don’t succeed, I will be happy to try it your way.

@gsmith: thank you for pointing out the -restart option. I stumbled across it in the documentation, but did not understand it at the time and I am still not sure:

I would like that Graylog, MongoDB and Elasticsearch starts, as soon as the server is booted. If the container stops with an error, I would like to now, so that I can follow it up. I the container is booted automatically, I might not be aware of errors, which crashes Graylog from time to time - if that happens at all.

Would you use

-restart: always

or would you use

-restart: unless-stopped

?

gsmith · December 20, 2022, 10:56pm

If your container/s stop or have issues, then you would need other software to help you get those alert or notifications. I belive docker has softerware for this purpose.

You could test this out by configuring one of those settings and rebooting the host.

schneich · January 2, 2023, 2:41pm

Just for the record, I ended up using this:

restart: unless-stopped

So far, no known issues with this configuration.

system · January 16, 2023, 2:42pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unhealthy Graylog docker image Graylog Central (peer support)	4	973	October 4, 2023
Couldn't read cluster health for indices [graylog_*] Graylog Central (peer support)	9	3083	March 12, 2019
Can't get docker graylog up and running Graylog Central (peer support)	3	7704	January 2, 2019
Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200) Graylog Central (peer support)	5	2542	April 25, 2020
Problems running graylog on google cloud VPS Graylog Central (peer support)	11	4564	August 19, 2017

Docker Graylog unhealthy

Related topics