Healthcheck fails for graylog container, GRAYLOG_HTTP_PUBLISH_URI doesn't get set in graylog.conf

Graylog runs and everything seems to be working fine, but if I issue a “docker ps” on the Host, I see that the Status is “unhealthy”.
I tried to run /health_check.sh step by step and check the variables when I noticed, in the script, the $check_url never gets set to my configured http_publish_uri. It stays empty the whole time.
I checked if the http_publish_uri is set in graylog.conf, but it isn’t:

cat data/config/graylog.conf | grep publish
#### HTTP publish URI
# The URI will be published in the cluster discovery APIs, so that other Graylog nodes will be able to find and connect to this Graylog node.
#http_publish_uri = http://192.168.1.1:9000/
# Default: $http_publish_uri

Even though I specified it in my docker-compose.yml:

      - GRAYLOG_HTTP_EXTERNAL_URI=https://host.domain.ch:9000/
      - GRAYLOG_HTTP_BIND_ADDRESS=127.0.0.1:9000
      - GRAYLOG_HTTP_PUBLISH_URI=https://host.domain.ch:9000/

Now I’m wondering why the http_publish_uri doesn’t get set in the config.

I’m running docker version 20.10.13, build a224086 and docker compose version 1.25.0 on Ubuntu 21.04.
I specified the Graylog image to " graylog/graylog:4.2-jre11"

If there are any logs that would help you, I’ll gladly provide them.
Thanks in advance

Hello && Welcome

I might be able to help.

From what I understand Graylog is working fine but your health check shows Unhealthy, is this correct?
If so, Could you explain more about this health check script (i.e. health_check.sh)?

I’m noticing that your using HTTPS with a FQDN? So I’m assuming your have certificates being used.
If this is correct?

Then the GRAYLOG_HTTP_BIND_ADDRESS is 127.0.0.1. Again I’m unsure about how this health script works but is it inside the contain or out side? I’m curious if the right ports are opened in the container to identify this health check process. Showing you full Docker file would be appreciated.
Do you see anything that may pertain to this issue in you logs?

EDIT: I forgot to ask, have you tried to test the following?

- GRAYLOG_HTTP_BIND_ADDRESS=127.0.0.1:9000

to

- GRAYLOG_HTTP_BIND_ADDRESS=0.0.0.0:9000

EDIT: I did some further research and found these posts. So I’m assuming this is the health_check script your referring to.

Aright,
So last night I installed Graylog Docker instance. To be honest it took a few hours to re-learn how to do this.

This is what I found out.
When running Docker-compose it used the docker-compose.yaml file, I searched in each one of these graylog.conf files shown below.

root@ansible:/usr/local/bin# locate graylog.conf
/var/lib/docker/overlay2/2bb622f895793e3ab0fed2992b78f0e8820acdfb5f6fabd74b02487bbd251aab/diff/usr/share/graylog/graylog.conf.example
/var/lib/docker/overlay2/2bb622f895793e3ab0fed2992b78f0e8820acdfb5f6fabd74b02487bbd251aab/diff/usr/share/graylog/data/config/graylog.conf
/var/lib/docker/overlay2/55bee51d8bf858c2cbbf2e5b529e360f8aa784eb2b0f4f0e4171168442d7e610/diff/usr/share/graylog/graylog.conf.example
/var/lib/docker/overlay2/730008e2d74a10460b66fa806da58f6d157329c31332e4d31347b6cafb3fdc13/diff/usr/share/graylog/graylog.conf.example
/var/lib/docker/overlay2/730008e2d74a10460b66fa806da58f6d157329c31332e4d31347b6cafb3fdc13/diff/usr/share/graylog/data/config/graylog.conf
/var/lib/docker/overlay2/aebfbb8e2fdcac2a68e0a3486f48d570889cb2f49eaee30ca51e14a4c45a8e97/diff/usr/share/graylog/data/config/graylog.conf
/var/lib/docker/overlay2/b3e30b82ceaf46e0f3f9e62457c3d00e8658c477617c29f92ad6bd57417c8a84/merged/usr/share/graylog/graylog.conf.example
/var/lib/docker/overlay2/b3e30b82ceaf46e0f3f9e62457c3d00e8658c477617c29f92ad6bd57417c8a84/merged/usr/share/graylog/data/config/graylog.conf
/var/lib/docker/overlay2/fa4c00ed13ce087eec51cc298b9b7eb6dfffb80b99e6c9efe3525584eb533d17/diff/usr/share/graylog/data/config/graylog.conf
/var/lib/docker/overlay2/fb4b4a0da0fa8774a28a5cac46fb39ebf62526660c1d53e89db1201e4a4b402e/diff/usr/share/graylog/graylog.conf.example
/var/lib/docker/volumes/147bd8051750fef8575cdad099efbe4597b0b187741f1c2240716c1286c6da18/_data/config/graylog.conf
/var/lib/docker/volumes/56cd6c48cae7fa81d242f5721e72c31db5a10522fde6487f3c4edaa7c02a7408/_data/config/graylog.conf
/var/lib/docker/volumes/7cb86d0f517371b620f7b06086b827dc0075090cf5d11eef4f126412540ccbcd/_data/config/graylog.conf
/var/lib/docker/volumes/80e2602ef0031860ab21ef66ba64ea426f1422cae1fafa9f7507b1ea27c2c624/_data/config/graylog.conf
/var/lib/docker/volumes/c42d2d0a627fca53e2032ae51a89f8df8943d792fda23bfdf95f7234dbafb4d6/_data/config/graylog.conf
/var/lib/docker/volumes/f6e06ada3d4cafa97fdc3895bf5aa861f3f764519aa77131bc935fcef31431c2/_data/config/graylog.conf
root@ansible:/usr/local/bin#

Every one of those files were the same. Then I understood, when the documentation stated " for testing and demo of Graylog".

So basically the reason that I noticed was graylog.conf file is the docker-compose.yaml.
Only thing I haven’t tested was creating my own container with out using Docker-compose.

As for the unhealthy, I think that is a configuration issue. Here is my Docker file and I’m not using https.

root@ansible:/usr/local/bin# cat docker-compose.yaml
version: '2'
services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongodb:
    image: mongo:4
  #DB in share for persistence
    volumes:
      - mongo_data:/data/db
  # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.6/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
    #data folder in share for persistence
    volumes:
      - es_data:/usr/share/elasticsearch/data
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:4.1
    #journal and config directories in local NFS share for persistence
    volumes:
      - graylog_journal:/usr/share/graylog/data/journal
    environment:
      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=pJod1TRZAckHmqM2oQPqX1qnLVJS99jHm2DuCux2Bpiuu2XLTZuyb2YW9eHiKLTifjy7cLpeWIjWgMtnwZf6Q79HW2nonDhN
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=ef92b778bafe771e89245b89ecbc08a44a4e166c06659911881f383d4473e94f
      - GRAYLOG_HTTP_BIND_ADDRESS=0.0.0.0:9000
      - GRAYLOG_HTTP_EXTERNAL_URI=http://10.8.8.8:9000/
      - GRAYLOG_TRANSPORT_EMAIL_WEB_INTERFACE_URL=http://10..8.8.8:9000
      - GRAYLOG_TRANSPORT_EMAIL_ENABLED=false
      - GRAYLOG_TRANSPORT_EMAIL_PORT=25
      - GRAYLOG_TRANSPORT_EMAIL_USE_AUTH=false
      - GRAYLOG_TRANSPORT_EMAIL_USE_TLS=false
      - GRAYLOG_TRANSPORT_EMAIL_USE_SSL=false
      - GRAYLOG_TRANSPORT_FROM_EMAIL=youremail@yourdomain.com
      - GRAYLOG_TRANSPORT_SUBJECT_PREFIX=[graylog]
    links:
      - mongodb:mongo
      - elasticsearch
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 8514:8514
      # Syslog UDP
      - 8514:8514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp
# Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
  mongo_data:
    driver: local
  es_data:
    driver: local
  graylog_journal:
    driver: local
root@ansible:/usr/local/bin#

Here is the web UI. You can see the redirect in the URL.

And last, here the output on docker ps command.

Conclusion:
Docker-compose used the YAML file. Unhealthy container is a configuration problem.
I could create another lab test on Docker, if you really want to use a graylog.conf file. I have a feeling its something like this command /w out Docker-compose.

docker run --link mongo --link elasticsearch \    
    -p 9000:9000 -p 12201:12201 -p 1514:1514 -p 5555:5555 \    
    -e GRAYLOG_HTTP_EXTERNAL_URI="http://127.0.0.1:9000/" \    
    -d graylog/graylog:<version_number>

Edit:
I found this section of the documentation which explain in greater detail. So yeah you can use “graylog.conf” file. Just need to reconfigure the environment.

Thank you very much for the time and effort you spent to help me.

The thing is, I had it working using only HTTP, just as you do now with docker telling the container is healthy. I then created my certificates and changed the URIs to fit HTTPS, which resulted in the unhealthy status.

I did try to change the Bind Address to 0.0.0.0:9000 and even the IP as well as the FQDN, but nothing worked.

My docker-compose.yml file looks like this now:

version: '2.1'
services:
{...}
# Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:4.2-jre11
    volumes:
      - graylog_data:/usr/share/graylog/data
      - ./certs:/etc/ssl/certs/graylog/
    environment:
      - TZ=Europe/Zurich
      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=secretpasswordforgraylog
      # Password: admin
      - GRAYLOG_TIMEZONE=Europe/Zurich
      - GRAYLOG_ROOT_TIMEZONE=Europe/Zurich
      - GRAYLOG_ROOT_PASSWORD_SHA2=35af527f0ab05beafebc6e341127f338c12edfc7479c07f7869721840665bcee
      - GRAYLOG_HTTP_EXTERNAL_URI=https://host.domain.ch:9000/
      - GRAYLOG_HTTP_BIND_ADDRESS=0.0.0.0:9000
      - GRAYLOG_HTTP_PUBLISH_URI=https://host.domain.ch:9000/
      - GRAYLOG_HTTP_ENABLE_CORS=true
      - GRAYLOG_HTTP_ENABLE_TLS=true
      - GRAYLOG_HTTP_TLS_CERT_FILE=/etc/ssl/certs/graylog/host.domain.ch.crt
      - GRAYLOG_HTTP_TLS_KEY_FILE=/etc/ssl/certs/graylog/host.domain.ch.key
    entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh
    links:
        - mongodb:mongo
        - elasticsearch
    restart: always
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 1514:1514
      # Syslog UDP
      - 1514:1514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp
# Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
  mongo_data:
    driver: local
  es_data:
    driver: local
  graylog_data:
    driver: local

You’re right, I could just use a custom configuration file where I will set the http_publish_uri manually, that just isn’t the smooth way of fixing the issue but rather a workaround in my opinion. But I might as well just do it.

By my understanding just the line

      - GRAYLOG_HTTP_PUBLISH_URI=https://host.domain.ch:9000/

should set the http_publish_uri in the ~/data/config/graylog.conf, but it does not, resulting in a fail of the /health_check.sh.

----- EDIT ----

I now put all the variables I need to set in a separate graylog.conf file, which replaces the default file. The Container still is listed as unhealthy, but docker logs shows me some warnings that might help, but I have to look into that.

2022-03-16 08:55:02,744 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call https://host.domain.ch:9000/api/system/metrics/multiple on node <39a013b4-eb50-4a0a-891b-7e089aff8858>: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

So I got it working now, there were a few problems with my setup.

Reason number one was because the Certificates were only internal, self signed certificates that couldn’t be handled by the java keystore. I could have put my CA in the keystore, but that would have been really messy.

The fix was to issue new Let’s Encrypt certificates that are issued by the CA of the company I’m working at. Still internal certificates, but a trusted CA.

I then reconfigured the files a bit, so now I don’t have to copy any certificates or replace the config file. All I need is the docker-compose.yml and the certificates at the location Let’s Encrypt puts them anyway.
docker-compose.yml:

version: '2.1'
services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongodb:
    image: mongo:4.2
    volumes:
      - mongo_data:/data/db
   # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
    volumes:
      - es_data:/usr/share/elasticsearch/data
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:4.2-jre11
    volumes:
      - graylog_data:/usr/share/graylog/data
      - /etc/letsencrypt/live/host.domain.ch/:/etc/ssl/certs/graylog/
      - /etc/letsencrypt/archive/host.domain.ch/:/etc/ssl/archive/host.domain.ch/
    environment:
      - TZ=Europe/Zurich
      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=secretpasswordforgraylog
      # Password: admin
      - GRAYLOG_TIMEZONE=Europe/Zurich
      - GRAYLOG_ROOT_TIMEZONE=Europe/Zurich
      - GRAYLOG_ROOT_PASSWORD_SHA2=35af527f0ab05beafebc6e341127f338c12edfc7479c07f7869721840665bcee
      - GRAYLOG_HTTP_EXTERNAL_URI=https://host.domain.ch:9000/
      - GRAYLOG_HTTP_BIND_ADDRESS=0.0.0.0:9000
      - GRAYLOG_HTTP_PUBLISH_URI=https://host.domain.ch:9000/
      - GRAYLOG_WEB_ENDPOINT_URI=https://host.domain.ch:9000/
      - GRAYLOG_HTTP_ENABLE_CORS=true
      - GRAYLOG_HTTP_ENABLE_TLS=true
      - GRAYLOG_HTTP_TLS_CERT_FILE=/etc/ssl/certs/graylog/fullchain.pem
      - GRAYLOG_HTTP_TLS_KEY_FILE=/etc/ssl/certs/graylog/privkey.pem
    entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh
    links:
        - mongodb:mongo
        - elasticsearch
    restart: always
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 1514:1514
      # Syslog UDP
      - 1514:1514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp
# Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
  mongo_data:
    driver: local
  es_data:
    driver: local
  graylog_data:
    driver: local
1 Like

Glad you fixed it,:+1:

NOTE: When stating this, I kind knew it was you certificates.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.