Unhealthy Graylog docker image

[hangmanandhide (M)]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
I came across graylog in my attempts to setup a syslog server for my unraid server. When I searched google, I found a github distro for instructions to setup not just a graylog server but also do it with the purpose of logging pfsense which I also use. I decided to follow these instructions because I would then get both.

The original distro is GitHub - lephisto/pfsense-analytics: Pfsense Analytics w/ Graylog, Elasticsearch, InfluxDB and Grafana fully dockerized for Firewall and DPI.. but the fork I used was from thenaturalwill (same link above but thenaturalwill instead of lephisto, restricted to 2 links cause im new) because this fork is the most recent maintained and fixed for log4j.

I have followed all steps and graylog comes up as unhealthy and doesnt let me hit the web portal. I tried following Docker Graylog unhealthy which seems to have a very similar issue, but I still can’t seem to get it to work and now when I do sudo docker ps, while mongo4.2 shows up as a container, it constantly says Restarting (132) X seconds ago as the status.

2. Describe your environment:

• OS Information:
  Debian GNU/Linux 11 (bullseye)

• Service logs, configurations, and environment variables:

Logs I see…

2023-09-15 21:49:42,090 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongo:27017
com.mongodb.MongoSocketException: mongo
	at com.mongodb.ServerAddress.getSocketAddresses(ServerAddress.java:211) ~[graylog.jar:?]
	at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:75) ~[graylog.jar:?]
	at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[graylog.jar:?]
	at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:128) ~[graylog.jar:?]
	at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:117) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
Caused by: java.net.UnknownHostException: mongo
	at java.net.InetAddress.getAllByName0(InetAddress.java:1281) ~[?:1.8.0_322]
	at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_322]
	at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_322]
	at com.mongodb.ServerAddress.getSocketAddresses(ServerAddress.java:203) ~[graylog.jar:?]
	... 5 more
2023-09-15 21:49:49,240 INFO : org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
2023-09-15 21:49:54,670 INFO : org.mongodb.driver.cluster - Exception in monitor thread while connecting to server mongo:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
	at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70) ~[graylog.jar:?]
	at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:128) ~[graylog.jar:?]
	at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:117) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_322]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_322]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_322]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_322]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_322]
	at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_322]
	at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:64) ~[graylog.jar:?]
	at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79) ~[graylog.jar:?]
	at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[graylog.jar:?]
	... 3 more
2023-09-15 21:50:20,019 INFO : org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
^C

My docker-compose.yml:

version: '3'

services:

  # MongoDB: hub_docker_com/_/mongo/
  mongodb:
    image: mongo:4.2
    volumes:
      - mongo_data:/data/db
    restart: always

  # Elasticsearch: www_elastic_co/guide/en/elasticsearch/reference/6.x/docker_html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.8
    # mem_limit: 4g
    restart: always
    volumes:
      - es_data:/usr/share/elasticsearch/data
    env_file:
      - ./elasticsearch.env
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - 9200:9200
    deploy:
     resources:
      limits:
       memory: 4g

  # Graylog: hub_docker_com/r/graylog/graylog/
  graylog:
    build:
      context: ./graylog/.
    volumes:
      - graylog_journal:/usr/share/graylog/data/journal
      - ./graylog/service-names-port-numbers.csv:/etc/graylog/server/service-names-port-numbers.csv
    env_file:
      - ./graylog.env
    entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh
    links:
      - mongodb:mongo
      - elasticsearch
    restart: always
    depends_on:
      - mongodb
      - elasticsearch
    healthcheck:
       test: curl --fail http://localhost:9000/ || exit 1
       interval: 40s
       timeout: 30s
       retries: 3
#       start_period: 60s
    ports:
      # Netflow
 - 2055:2055/udp
      # Syslog Feed
      - 5442:5442/udp
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 1514:1514
      # Syslog UDP
      - 1514:1514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp

  # Kibana : www_elastic_co/guide/en/kibana/6.8/index_html
  kibana:
    image: docker.elastic.co/kibana/kibana:7.17.8
    entrypoint: ["echo", "Service Kibana disabled"]
    env_file:
      - kibana.env
    depends_on:
      - elasticsearch
    ports:
      - 5601:5601
  cerebro:
    image: lmenezes/cerebro
#    entrypoint: ["echo", "Service cerebro disabled"]
    ports:
      - 9001:9000
    links:
      - elasticsearch
    depends_on:
      - elasticsearch
    logging:
      driver: "json-file"
      options:
        max-size: "100M"

  influxdb:
#    image: 'influxdb:1.8.4-alpine'
    image: 'influxdb:2.7-alpine'
    env_file:
      - ./influxdb.env
    ports:
      - 8086:8086
    volumes:
      - influxdb:/var/lib/influxdb
    logging:
      driver: "json-file"
      options:
        max-size: "100M"

  grafana:
    image: grafana/grafana-enterprise:latest
    env_file:
      - ./grafana.env
    ports:
      - 3000:3000
    volumes:
      - grafana:/var/lib/grafana
      - ./grafana/provisioning/:/etc/grafana/provisioning
    links:
      - elasticsearch
      - influxdb
    depends_on:
      - elasticsearch
      - influxdb
    logging:
      driver: "json-file"
      options:
        max-size: "100M"

# Volumes for persisting data, see docs_docker_com/engine/admin/volumes/volumes/
volumes:
  mongo_data:
    driver: local
  es_data:
    driver: local
  graylog_journal:
    driver: local
  grafana:
    driver: local
  influxdb:
    driver: local

3. What steps have you already taken to try and solve the problem?

As mentioned, I have tried searching and troubleshooting even through this site but I am far outside my individual ability. I am not versed in anything beyond following instructions on the screen and simple troubleshooting. I don’t use docker on a daily basis and I certainly don’t know how to solve issues beyond googling.

4. How can the community help?

If someone could help me figure out how to get graylog healthy and responding to a webportal request so I can get it running I would be so incredibly grateful. I don’t know why I can’t get this to run.

[drewmiranda-gl (Drew Miranda)]

On the surface it appears that the graylog docker container isn’t configured to connect to mongo and is falling back to the default setting. When that happens graylog attempts to connect to mongo at 127.0.0.1:27017. Since mongodb is not running locally in the same container as graylog it will fail to connect.

For a reference of example parameters, check out https://github.com/Graylog2/docker-compose/blob/main/open-core/docker-compose.yml

There are also some additional concerns or issues:

• Its building a custom docker image using the build.context docker compose property and building https://github.com/lephisto/pfsense-analytics/blob/master/Docker/graylog/Dockerfile
• The above is using Graylog 4 which is a few years old at this point
• An unsupported version of Elasticsearch is being used

Docker is designed to be modular so my recommendation is to use our example Graylog docker compose file for running graylog. Using docker volumes and bind mounts its possible to add any files needed into graylog so the custom build step is unnecessary.

You may also reach out to the owner of the docker compose file you shared to have them both update and fix the issues.

Hope that helps.

[hangmanandhide (M)]

Hi there,

Because it took so long for me to get this post unmarked out of spam I had to dig deep and really get gritty to get past where I was stuck. Most of my help came care of working with HalianElf on the Unraid Community discord (props for being a pal). Here’s everything we did/figured out in case someone else out there in the worldwide web is or will at somepoint try to do these very same things and end up stuck like I was.

1. MongoDB container wasnt running! Was getting a critical error that the docker image that it was pulling couldn’t deploy because my system wasnt compatible with it. IE my rPi4-8gb cpu isnt compatible with v4.

I eventually found this- GitHub - themattman/mongodb-raspberrypi-docker: Docker Images for MongoDB on Raspberry Pi .

Clone it locally and build it, make sure to tag it something (I chose mongounofficial:latest)

Then I edited my docker compose file and commented out the original image and put the tag in. I then docker down and docker up’d the compose file to get everything up and running.

This took care of the issues I had graylog. However there ended up being more.

2- influxDB. The original compose file calls for 1.8.4-alpine. Latest didn’t work for me as cerebro only wants a 1.X version. influxdb:1.5.1 worked for me… or at least enough for me to continue through the instructions in cerebro, which takes me to part 3.

3- cerebro. The listed ‘image: lmenezes/cerebro’ didn’t build for me. I had to clone and build a repo separately. GitHub - lmenezes/cerebro-docker: official cerebro docker image . Make sure to tag it similar to the instructions above for mongo. Repeat same steps in compose file but cerebro section and you should have no problem deploying with compose up.

Final thoughts:

I was able to get it all to work–ish. I am having issues with grafana right now. The DPI tab is working (except the geolocation data/map for some reason) but I cant get the firewall dashboard to work. I dont know how to troubleshoot this or fix it and Im at another–possibly immovable-- dead end. I dont know if its influxdb, if its grafana, or what. I am way outside my comfort zone. I hope this information helps someone. Always grateful to find resources that get me past whatever problems I have with my own projects.

Good luck.

[gsmith]

Hey @hangmanandhide

Notice you use Cerebro also but the Docker version. This is off topic i think but did you state you had to build Cerebro-Docker image? If so by chance did you post anything about the build in Github for that?
Just curious. I have been using the reg cerebro install for a long time on Ubuntu and was think about going the docker route.