Graylog container-persistence not working on absolute path

SofaKingBoring · June 17, 2022, 9:46am

Hello Community,

I got a problem with making my Docker-Containter-Setup persistent.
I put in a lot of effort solving it myself, but this is my first real docker home network project, so bare with me please

Description of your problem

My only problem is mounting volumes to specific, absolute paths only for the graylog image.
For Elasticsearch and Mongo it does work

Description of steps you’ve taken to attempt to solve the issue

When it didn’t work I started with making sure it worked without mounting volumes from the containers. It did.

Then I tested for each of the 3 images seperately. Narrowed it down to “graylog/graylog”

Then I tested using volume in docker’s default directory by defining volume like this:
- graylog_data:/usr/share/graylog/data/
instead of
- /dockvol/graylog/graylog_data:/usr/share/graylog/data
That worked also and data was written in `/var/lib/docker/volumes/graylog_graylog_data/_data

Then I compared permissions of
/var/lib/docker/volumes/graylog_graylog_data/_data
and
/dockvol/graylog/graylog_data/
with no success, although I believe that the problem must be somewhere here

Finally I added
"GRAYLOG_SERVER_JAVA_OPTS=-Dlog4j2.debug=true" to the graylog-service to get some Debugging info.

Environmental information

Operating system information

Ubuntu Server

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal

Package versions

REPOSITORY                                          TAG       IMAGE ID       CREATED         SIZE
graylog/graylog                                     4.2       3d389cc20dde   6 weeks ago     473MB
mongo                                               4.2       5db1c04860da   6 weeks ago     388MB
docker.elastic.co/elasticsearch/elasticsearch-oss   7.10.2    b313026e6fbd   17 months ago   699MB

Service logs, configuration, and environment variables

docker-compose.yml:

version: '3'
services:
    # MongoDB: https://hub.docker.com/_/mongo/
    mongo:
      image: mongo:4.2
      networks:
        - graylog
      volumes:
        - /dockvol/graylog/mongodb:/data/db
    # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/docker.html
    elasticsearch:
      image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
      volumes:
        - /dockvol/graylog/es_data:/usr/share/elasticsearch/data
      environment:
        - http.host=0.0.0.0
        - transport.host=localhost
        - network.host=0.0.0.0
        - "ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true -Xms512m -Xmx512m"
      ulimits:
        memlock:
          soft: -1
          hard: -1
      networks:
        - graylog
    # Graylog: https://hub.docker.com/r/graylog/graylog/
    graylog:
      image: graylog/graylog:4.2
      user: "1100"
      volumes:
        #- graylog_data:/usr/share/graylog/data/
        - /dockvol/graylog/graylog_data:/usr/share/graylog/data
      environment:
        - GRAYLOG_PASSWORD_SECRET=XXX
        # Password: admin
        - GRAYLOG_ROOT_PASSWORD_SHA2=XXX
        - GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
        - "GRAYLOG_SERVER_JAVA_OPTS=-Dlog4j2.debug=true"
      entrypoint: /usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh
      networks:
        - graylog
      restart: always
      depends_on:
        - mongo
        - elasticsearch
      ports:
        # Graylog web interface and REST API
        - 9000:9000
        # Syslog TCP
        - 1514:1514
        # Syslog UDP
        - 1514:1514/udp
        # GELF TCP
        - 12201:12201
        # GELF UDP
        - 12201:12201/udp
networks:
    graylog:
      driver: bridge
volumes:
    graylog_data:

directory permissions:

root@asdf# ll /var/lib/docker/volumes/graylog_graylog_data/_data/
total 36
drwxr-x--- 9 graylog graylog 4096 Jun 15 22:33 ./
drwx-----x 3 root    root    4096 Jun 15 22:33 ../
drwxr-x--- 2 graylog graylog 4096 Jun 15 22:33 config/
drwxr-xr-x 2 graylog graylog 4096 Jun 15 22:33 contentpacks/
drwxr-x--- 2 graylog graylog 4096 May  4 18:18 data/
drwxr-x--- 3 graylog graylog 4096 Jun 17 10:37 journal/
drwxr-xr-x 2 graylog graylog 4096 Jun 17 10:37 libnative/
drwxr-x--- 2 graylog graylog 4096 May  4 18:18 log/
drwxr-x--- 2 graylog graylog 4096 May  4 18:18 plugin/

root@asdf ll /dockvol/graylog/graylog_data/
total 28
drwxr-xr-x 7 graylog graylog 4096 Jun 15 22:15 ./
drwxr-xr-x 5 root    root    4096 Jun 15 22:39 ../
drwxr-xr-x 2 graylog graylog 4096 Jun 15 22:30 config/
drwxr-xr-x 2 graylog graylog 4096 Jun 15 22:15 contentpacks/
drwxr-xr-x 2 graylog graylog 4096 Jun 15 22:15 journal/
drwxr-xr-x 2 graylog graylog 4096 Jun 15 22:15 log/
drwxr-xr-x 2 graylog graylog 4096 Jun 15 22:15 plugin/

users:

root@asdf# grep graylog /etc/passwd
graylog:x:1100:1100:Graylog,,,:/home/graylog:/bin/bash

root@asdf# grep graylog /etc/group
graylog:x:1100:

and (imho) the relevant error message:

graylog_1        | DEBUG StatusLogger Catching
graylog_1        |  java.io.FileNotFoundException: /usr/share/graylog/data/config/log4j2.xml (No such file or directory)
graylog_1        |      at java.io.FileInputStream.open0(Native Method)
graylog_1        |      at java.io.FileInputStream.open(FileInputStream.java:195)
graylog_1        |      at java.io.FileInputStream.<init>(FileInputStream.java:138)
graylog_1        |      at org.apache.logging.log4j.core.config.ConfigurationFactory.getInputFromString(ConfigurationFactory.java:370)
graylog_1        |      at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:512)
graylog_1        |      at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:498)
graylog_1        |      at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:422)
graylog_1        |      at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:323)
graylog_1        |      at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:695)
graylog_1        |      at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:716)
graylog_1        |      at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:270)
graylog_1        |      at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:155)
graylog_1        |      at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
graylog_1        |      at org.apache.logging.log4j.LogManager.getContext(LogManager.java:196)
graylog_1        |      at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:137)
graylog_1        |      at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:55)
graylog_1        |      at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:47)
graylog_1        |      at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:33)
graylog_1        |      at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:363)
graylog_1        |      at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
graylog_1        |      at org.graylog2.bootstrap.CmdLineTool.<clinit>(CmdLineTool.java:106)
graylog_1        |      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
graylog_1        |      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
graylog_1        |      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
graylog_1        |      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
graylog_1        |      at com.github.rvesse.airline.parser.ParserUtil.createInstance(ParserUtil.java:39)
graylog_1        |      at com.github.rvesse.airline.DefaultCommandFactory.createInstance(DefaultCommandFactory.java:25)
graylog_1        |      at com.github.rvesse.airline.parser.ParserUtil.createInstance(ParserUtil.java:94)
graylog_1        |      at com.github.rvesse.airline.parser.ParseResult.getCommand(ParseResult.java:114)
graylog_1        |      at com.github.rvesse.airline.parser.command.CliParser.parse(CliParser.java:54)
graylog_1        |      at com.github.rvesse.airline.Cli.parse(Cli.java:127)
graylog_1        |      at com.github.rvesse.airline.Cli.parse(Cli.java:113)
graylog_1        |      at org.graylog2.bootstrap.Main.main(Main.java:43)
graylog_1        | ERROR StatusLogger Reconfiguration failed: No configuration found for '70dea4e' at 'null' in 'null'
graylog_1        | DEBUG StatusLogger Shutdown hook enabled. Registering a new one.

Thanks for anyone who took the effort to read and tries to help.

Regards
Chris

gsmith · June 17, 2022, 10:00pm

Hello @SofaKingBoring && Welcome

I’m also playing around with Docker/Docker-Compose. Just learning the in’s & Outs of this software.

I have not used the absolute path for a image, I thought Docker took care of that already? After working with this setup the one thing I like best was the quick way of upgrading the software by adjusting the image from the yaml file then restarting. From the logs it looks like Graylog can not find
the log4j2.xml file. this should be in the home directory/container.

Not sure what the issue could be but I can show you my compose file, perhaps that might help.

This is on Ubuntu 20.0 just for testing out Enterprise stuff.

yaml file

version: '3'
services:
   # MongoDB: https://hub.docker.com/_/mongo/
  mongodb:
    image: mongo:4.4
    network_mode: bridge
   # DB in share for persistence
    volumes:
      - mongo_data:/data/db
   # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.6/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2-amd64
    # image: opensearchproject/opensearch:1.3.2
    network_mode: bridge
    #data folder in share for persistence
    volumes:
      - es_data:/usr/share/elasticsearch/data
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      #- network.publish_host=10.200.6.28
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
   # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog-enterprise:4.3-jre11
    network_mode: bridge
    dns:
      - 8.8.8.8
      - 8.8.4.4
   # journal and config directories in local NFS share for persistence
    volumes:
       - graylog_journal:/usr/share/graylog/data/journal
       - graylog_bin:/usr/share/graylog/bin
       - graylog_data:/usr/share/graylog/data/config
       - graylog_log:/usr/share/graylog/data/log
       - graylog_plugin:/usr/share/graylog/data/plugin
       - graylog_content:/usr/share/graylog/data/contentpacks
      # Mount local configuration directory into Docker container
      
    environment:
      # Container time Zone
      - TZ=America/Chicago
      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=pJod1TRZpiuu2XLTZuifjy7cLpetnwZf6Q79HW2nonDhN
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=ef92b778bafe771e89911881f383d4473e94f
      - GRAYLOG_HTTP_BIND_ADDRESS=0.0.0.0:9000
      - GRAYLOG_HTTP_EXTERNAL_URI=http://10.10.10.10:9000/
      - GRAYLOG_ROOT_TIMEZONE=America/Chicago
      - GRAYLOG_ROOT_EMAIL=greg.smith@enseva.com
      - GRAYLOG_HTTP_PUBLISH_URI=http://10.10.10.10:9000/
      - GRAYLOG_TRANSPORT_EMAIL_PROTOCOL=smtp
      - GRAYLOG_HTTP_ENABLE_CORS=true
      - GRAYLOG_TRANSPORT_EMAIL_WEB_INTERFACE_URL=http://10.10.10.10:9000/
      - GRAYLOG_TRANSPORT_EMAIL_HOSTNAME=10.10.10.10
      - GRAYLOG_TRANSPORT_EMAIL_ENABLED=true
      - GRAYLOG_TRANSPORT_EMAIL_PORT=25
      - GRAYLOG_TRANSPORT_EMAIL_USE_AUTH=false
      - GRAYLOG_TRANSPORT_EMAIL_USE_TLS=false
      - GRAYLOG_TRANSPORT_EMAIL_USE_SSL=false
      - GRAYLOG_TRANSPORT_FROM_EMAIL=root@localhost
      - GRAYLOG_TRANSPORT_SUBJECT_PREFIX=[graylog]
      - GRAYLOG_REPORT_DISABLE_SANDBOX=true
      - GRAYLOG_REPORT_RENDER_URI=http://10.10.10.10:9000
      # - GRAYLOG_REPORT_USER=graylog-report
      - GRAYLOG_REPORT_RENDER_ENGINE_PORT=9515
    links:
      - mongodb:mongo
      - elasticsearch
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 8514:8514
      # Elasticsearch
      - 9200:9200
      - 9300:9300
      # Syslog UDP
      - 8514:8514/udp
      # GELF TCP
      #- 12201:12201
      # GELF UDP
      #- 12201:12201/udp
      # Reports
      - 9515:9515
      - 9515:9515/udp
      # beats
      - 5044:5044
      # email
      - 25:25
      - 25:25/udp
      # web
      - 80:80
      - 443:443
      - 21:21
      # Forwarder
      - 13302:13302
      - 13301:13301
      # keycloak
      - 8443:8443
      # packetbeat
      - 5055:5055
#Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
  mongo_data:
    driver: local
  es_data:
    driver: local
  graylog_journal:
 driver: local
  graylog_bin:
    driver: local
  graylog_data:
    driver: local
  graylog_log:
    driver: local
  graylog_plugin:
    driver: local
  graylog_content:
    driver: local

REPOSITORY                                          TAG            IMAGE ID       CREATED         SIZE
graylog/graylog-enterprise                          4.3-jre11      ec6db44b0465   4 weeks ago     841MB
mongo                                               4.4            aa61e4b09cef   6 weeks ago     438MB
docker.elastic.co/elasticsearch/elasticsearch-oss   7.10.2-amd64   b313026e6fbd   17 months ago   699MB

The rest looks like you post ( passwd, etc…)

SofaKingBoring · June 18, 2022, 4:34pm

Yes it’s true that it is missing the log4j2.xml, but it should be writing it there at startup.
At least that’s what’s happening in the “default” folders /var/lib/docker, if I don’t use the absolute path as given above.

If noone else has an idea I will first try to get 4.3 running with opensearch and then get back to this issue.
I still hope someone will have an insight here

system · July 2, 2022, 4:34pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Persistence not working Graylog Central (peer support) docker	2	495	April 2, 2023
Docker persistent volume issue Graylog Central (peer support)	1	1361	May 5, 2021
Location of Roles, Users, and Streams Graylog Central (peer support)	3	527	December 26, 2017
Can't get docker graylog up and running Graylog Central (peer support)	3	7347	January 2, 2019
Problems with Graylog Docker image Graylog Central (peer support)	28	6500	August 11, 2017