Change from Elasticsearch to opensearch: Node ID problem

Dear community,

I am running Graylog 5.2 with Elasticsearch in a Docker Stack. Everything is fine.
Last night, I wanted to upgrade to Graylog 6.0 and make the change to opensearch.

Unfortunately, I cannot get it running.

  • How can I verify, if the containers can reach each other? “ping” is not available within the graylog-container.
  • When googling, this seems to be an old issue, but I did not find a working solution. How do I debug this?

I see this error log in the graylog container:

wait-for-it: waiting 15 seconds for elasticsearch:9200

wait-for-it: timeout occurred after waiting 15 seconds for elasticsearch:9200

2024-05-11T21:19:54.598201098Z main ERROR Reconfiguration failed: No configuration found for '50a4e294' at 'null' in 'null'

23:19:54.806 [main] INFO  org.graylog2.featureflag.ImmutableFeatureFlagsCollector - Following feature flags are used: {default properties file=[frontend_hotkeys=on, field_types_management=on, cloud_inputs=on, data_tiering_cloud=off, scripting_api_preview=on, composable_index_templates=off, search_filter=on, preflight_web=on, data_node_migration=off, instant_archiving=off]}

23:19:54.813 [main] ERROR org.graylog2.bootstrap.CmdLineTool - Couldn't load configuration: Properties file /usr/share/graylog/data/config/graylog.conf doesn't exist!

my compose.yaml

# version: as May 2024 https://docs.docker.com/compose/compose-file/

services:
# MongoDB: https://hub.docker.com/_/mongo/
  mongodb:
    image: mongo:6.0.14
    container_name: mongodb
    hostname: mongodb
    environment:
      - TZ=Europe/Zurich
    restart: unless-stopped
    #DB in share for persistence
    volumes:
      - type: bind
        source: /home/uadmin/Docker/Graylog6/mongo_data/mongo_db
        target: /data/db
      - type: bind
        source: /home/uadmin/Docker/Graylog6/mongo_data/mongo_configdb
        target: /data/configdb
    deploy:
      resources:
        limits:
          memory: 500mb
    networks:
      graylog_backend:
        ipv4_address: 10.10.10.13



  opensearch:
    image: opensearchproject/opensearch:2.12.0
    container_name: opensearch
    hostname: opensearch
    volumes:
      - /home/uadmin/Docker/Graylog6/opensearch-data:/usr/share/opensearch/data
    environment:
      - "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g"
      - "bootstrap.memory_lock=true"
      - "discovery.type=single-node"
      - "action.auto_create_index= false"
      - "plugins.security.ssl.http.enabled=false"
      - "plugins.security.disabled=true"
      # Can generate a password for `OPENSEARCH_INITIAL_ADMIN_PASSWORD` using a linux device via:
      # tr -dc A-Z-a-z-0-9_@#%^-_=+ < /dev/urandom  | head -c${1:-32}
      - "OPENSEARCH_INITIAL_ADMIN_PASSWORD=lOULLLfs2h3TKicWH7tVAOaP4qvNu3UcYA7BfCxxxx"
      - TZ=Europe/Zurich
    ulimits:
      memlock:
        hard: -1
        soft: -1
      nofile:
        soft: 65536
        hard: 65536
    restart: unless-stopped
    networks:
      graylog_backend:
        ipv4_address: 10.10.10.14





# Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:6.0
    container_name: graylog
    hostname: graylog
    domainname: internal
    #journal and config directories in local NFS share for persistence
    volumes:
      - /home/uadmin/Docker/Graylog6/graylog_data:/usr/share/graylog/data
    entrypoint: "/usr/bin/tini -- wait-for-it elasticsearch:9200 --  /docker-entrypoint.sh"
    environment:
      - GRAYLOG_NODE_ID_FILE= "/usr/share/graylog/data/config/node-id"
      - GRAYLOG_HTTP_BIND_ADDRESS= "192.168.60.12:9000"
      - GRAYLOG_ELASTICSEARCH_HOSTS= "http://opensearch:9200"
      - GRAYLOG_MONGODB_URI= "mongodb://mongodb:27017/graylog"
      # To make reporting (headless_shell) work inside a Docker container
      - GRAYLOG_REPORT_DISABLE_SANDBOX= "true"
      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET= "y9H2-muMO-qtAS-xxxx"
      # Password: "admin"
      - GRAYLOG_ROOT_PASSWORD_SHA2= "52993186fc321cf6969fc160d99ed41869703000868814d30deddff4e795xxxx"
      - GRAYLOG_HTTP_EXTERNAL_URI= "http://192.168.60.3:9000/"
      - TZ=Europe/Zurich
    deploy:
      resources:
        limits:
          memory: 2gb
    networks:
      macvlan60:
        ipv4_address: 192.168.60.13
      graylog_backend:
        ipv4_address: 10.10.10.12
    links:
      - mongodb:mongo
      - opensearch
    restart: unless-stopped
    depends_on:
      mongodb:
        condition: service_started
      opensearch:
        condition: service_started
    ports:
      # Graylog web interface and REST API
      - "9000:9000/tcp"
      # Beats
      - "5044:5044/tcp"
      # Syslog TCP
      - "5140:5140/tcp"
      - "1514:1514" #(e.g. Unifi)
      - "1515:1515" #(e.g. Tasmota)
      # Syslog UDP
      - "5140:5140/udp"
      # GELF TCP
      - "12201:12201/tcp"
      # GELF UDP
      - "12201:12201/udp"
      # Forwarder data
      - "13301:13301/tcp"
      # Forwarder config
      - "13302:13302/tcp"




# Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
  mongo_db:
  mongo_configdb:
  opensearch-data:
  graylog_data:
    driver: local


# Network specifications
networks:
  macvlan60:
    external: true
  graylog_backend:
    internal: true
    ipam:
      driver: default
      config:
        - subnet: "10.10.10.0/24"

Any help is very much appreciated!

Update: I followed this chapter
https://go2docs.graylog.org/current/downloading_and_installing_graylog/docker_installation.htm#Configuration

and got the missing graylog.conf from there.
And I had to map the port 9200 for the opensearch container.

Now, I have this error message:

wait-for-it: waiting 15 seconds for opensearch:9200

wait-for-it: opensearch:9200 is available after 0 seconds

2024-05-12 01:05:29,261 INFO : org.graylog2.featureflag.ImmutableFeatureFlagsCollector - Following feature flags are used: {default properties file=[frontend_hotkeys=on, field_types_management=on, cloud_inputs=on, data_tiering_cloud=off, scripting_api_preview=on, composable_index_templates=off, search_filter=on, preflight_web=on, data_node_migration=off, instant_archiving=off]}

2024-05-12 01:05:29,466 ERROR: org.graylog2.bootstrap.CmdLineTool - Invalid configuration

com.github.joschi.jadconfig.ValidationException: Parent path "/usr/share/graylog/data/config for Node ID file at "/usr/share/graylog/data/config/node-id" is not a directory

	at org.graylog2.Configuration$NodeIdFileValidator.validate(Configuration.java:572) ~[graylog.jar:?]

	at org.graylog2.Configuration$NodeIdFileValidator.validate(Configuration.java:560) ~[graylog.jar:?]

	at com.github.joschi.jadconfig.JadConfig.validateParameter(JadConfig.java:227) ~[graylog.jar:?]

	at com.github.joschi.jadconfig.JadConfig.processClassFields(JadConfig.java:143) ~[graylog.jar:?]

	at com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:101) ~[graylog.jar:?]

	at org.graylog2.bootstrap.CmdLineTool.processConfiguration(CmdLineTool.java:505) [graylog.jar:?]

	at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:293) [graylog.jar:?]

	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:267) [graylog.jar:?]

	at org.graylog2.bootstrap.Main.main(Main.java:55) [graylog.jar:?]

Any ideas?
I would expect it to create a random node-id?

I have spend the last night on this and I am stuck.

I still keep this error:

2024-05-12 09:49:44,403 ERROR: org.graylog2.bootstrap.CmdLineTool - Invalid configuration

com.github.joschi.jadconfig.ValidationException: Parent path "/usr/share/graylog/data/config for Node ID file at "/usr/share/graylog/data/config/node-id" is not a directory

	at org.graylog2.Configuration$NodeIdFileValidator.validate(Configuration.java:572) ~[graylog.jar:?]

	at org.graylog2.Configuration$NodeIdFileValidator.validate(Configuration.java:560) ~[graylog.jar:?]

	at com.github.joschi.jadconfig.JadConfig.validateParameter(JadConfig.java:227) ~[graylog.jar:?]

	at com.github.joschi.jadconfig.JadConfig.processClassFields(JadConfig.java:143) ~[graylog.jar:?]

	at com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:101) ~[graylog.jar:?]

	at org.graylog2.bootstrap.CmdLineTool.processConfiguration(CmdLineTool.java:505) [graylog.jar:?]

	at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:293) [graylog.jar:?]

	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:267) [graylog.jar:?]

	at org.graylog2.bootstrap.Main.main(Main.java:55) [graylog.jar:?]

The user rights for the persistant folder on my server look like this:

uadmin@ubuntu:~/Docker/Graylog6/graylog_data/config$ ls -lah
total 48K
drwxrwxrwx 2 graylog graylog 4.0K May 12 09:47 .
drwxrwxrwx 7 graylog graylog 4.0K May 11 23:07 ..
-rwxrwxrwx 1 graylog graylog  34K May 12 00:56 graylog.conf
-rwxrwxrwx 1 graylog graylog 2.5K May 12 00:56 log4j2.xml

Graylog is able to read the graylog.conf, but it does not write the node-id file.

If I remove an persistence from the graylog container, I still get the same error. Is there something wrong with image: graylog/graylog:6.0?

for the record: I finally got it up and running. I will sumarize everything and post a link.

Hey @schneich

Judging from the error

main] ERROR org.graylog2.bootstrap.CmdLineTool - Couldn't load configuration: Properties file /usr/share/graylog/data/config/graylog.conf doesn't exist!

Graylog is looking for the configuration file. By chance did you have this ocnfigured already.

Hi @gsmith,

thank you for your feedback. Yes, this was one of my errors.

  • It started with Graylog not being able to reach Opensearch.
  • Then, I realized that my graylog.conf was missing.
  • Then, the environment variable for the node-id file did not work.
  • In between, I had several issues with folder access permissions.

Here is a brief summary of my findings and a working solution.

Please allow me a followup question: is there a guide on how to migrate from Elasticsearch to Opensearch? And MongoDB 5 to MongoDB 6?

There is a section in the docs:
https://go2docs.graylog.org/5-2/planning_your_deployment/upgrading_to_opensearch.htm?tocpath=Upgrading%20Graylog|OpenSearch|Moving%20from%20Elasticsearch%20to%20OpenSearch|_____0

Best to start a new topic if you are having issues with migration.

Hi @patrickmann ,

Thank you. I have seen that page. It explains about elasticsearch to opensearch, but does not mention mongodb… I might have found other sources for that. If it works, I am happy to share my procedures. :slight_smile:

for the record: this is how I finally upgraded from Graylog 5.2 to 6.0.1:

2 Likes