Upgrade from Graylog from 5.0 to 6.3.1 and all our user accounts are gone

Graylog Central (peer support)
,
1

I just upgraded our company’s Graylog server from 5.0 to 6.3.1 and migrating from ElasticSearch to OpenSearch. The server is healthy and everything was in order until 1 days later I checked the server and all our users accounts were gone :cry:, and OpenSearch raised IndexError below:

“While retrieving data for this widget, the following error(s) occurred:
OpenSearch exception [type=index_not_found_exception, reason=no such index ].”

image
image1826×557 9.94 KB

I managed to fix the indices followed this but I still doesn’t understand why all our accounts is suddenly gone. I keep persistent Volume from previous Graylog and Mongo containers. Graylog is running in Docker container on Elastic Cloud Server.

System info:

  1. Old Graylog Ver
  • Graylog 5.0
  • Mongo: 5.0.13
  • Elastic Search: 7.10.2
  1. Upgraded Graylog Ver:
  • Graylog 6.3.1
  • Mongo: 7.0.22
  • OpenSearch: 2.15.0

OS: Ubuntu 22.04

Docker: Docker version 27.5.1, build 27.5.1-0ubuntu3~22.04.2

Here are the steps I made during the upgrade:

  • Backup Graylog Config file and Mongo data
  • Closed Graylog, Mongo, ElasticSearch Container: docker-compose -f docker-compose.yml down
  • Upgrade MongoDB 5.0 > 6.0: docker-compose -f docker-compose.yml -f docker-compose.override_mongo.yml up -d
  • Check Graylog Stream and Users
  • Change feature compatibility from 5.0 to 6.0: docker exec -it <mongo_container> mongosh
    • db.adminCommand({ setFeatureCompatibilityVersion: "6.0" })
  • Shut down all containers, then upgrade to Mongo 7.0, OpenSearch, Graylog 6.3.1: docker compose -f docker-compose.graylog6x.yml up -d
  • Stop OpenSearch, Graylog before upgrading to setting feature compataibility to 7.0
    • docker exec -it <mongo_container> mongosh
    • db.adminCommand({ setFeatureCompatibilityVersion: "7.0" })
    • db.adminCommand({ setFeatureCompatibilityVersion: "7.0" , confirm:true})
  • Rebuild the file
    docker compose -f docker-compose.graylog6x.yml up -d

The server was working fine and all our user accounts and streams were still intact at that time. I don’t know what happened? Is there any way to restored user accounts?
Here’s my docker-compose.yml file

services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongo:
    image: mongo:7.0.22
    restart: always
    networks:
      - graylog
    volumes:
      - mongodb_data:/data/db
      - mongodb_config:/data/configdb
    ports:
      - "27017:27017"

  # Open Search https://hub.docker.com/r/opensearchproject/opensearch/tags
  opensearch:
    image: opensearchproject/opensearch:2.15.0
    restart: always
    environment:
      - "bootstrap.memory_lock=true"
      - "discovery.type=single-node"
      - "action.auto_create_index= false"
      - "plugins.security.disabled=true"
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      #- "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g"
      # Required: Can generate a password for `OPENSEARCH_INITIAL_ADMIN_PASSWORD` using a linux device via:
      # tr -dc A-Z-a-z-0-9_@#%^-_=+ < /dev/urandom | head -c${1:-32}
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
    networks:
      - graylog
    # Configure persistent volume
    volumes:
      - opensearch_data:/usr/share/opensearch/data
    ports:
      - "9200:9200" #Required for access to the OpenSearch API.
      - "9300:9300" # Required for OpenSearch cluster communication. (Thus port is only required if you use self-managed OpenSearch

  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:6.3.1
    restart: always
    depends_on:
      - mongo
      - opensearch
    environment:
      # Password (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=${GRAYLOG_PASSWORD_SECRET}
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=${GRAYLOG_ROOT_PASSWORD_SHA2}
      - GRAYLOG_HTTP_EXTERNAL_URI=http://127.0.0.1:9000/
      - GRAYLOG_ELASTICSEARCH_HOSTS=http://opensearch:9200
      - GRAYLOG_MONGODB_URI=mongodb://mongo:27017/graylog
    networks:
      - graylog

    volumes:
      - graylog_data:/usr/share/graylog/data
    ports:
      - "5044:5044/tcp"   # Beats
      - "5140:5140/udp"   # Syslog
      - "5140:5140/tcp"   # Syslog
      - "5555:5555/tcp"   # RAW TCP
      - "5555:5555/udp"   # RAW UDP
      - "9000:9000/tcp"   # Server API
      - "12201:12201/tcp" # GELF TCP
      - "12201:12201/udp" # GELF UDP
      #- "10000:10000/tcp" # Custom TCP port
      #- "10000:10000/udp" # Custom UDP port
      - "13301:13301/tcp" # Forwarder data
      - "13302:13302/tcp" # Forwarder config

# Keep the existing volume for Graylog and MongoDB
volumes:
  graylog_data:
    external: true
    name: ${GRAYLOG_VOLUME_NAME}

  mongodb_data:
    external: true
    name: ${MONGODB_DATA_VOLUME_NAME}

  mongodb_config:
    external: true
    name: ${MONGODB_CONFIG_VOLUME_NAME}

  opensearch_data:

networks:
  graylog:
    driver: bridge
2

Hi @vy-kauppinen,

In your docker compose file, I don’t see any security and auth related configuration for mongodb. Is it possible that it’s running without any auth, opened to the outside world? I’ve seen similar problem before: Could not retrieve index sets. No default index set configured. This is a bug! · Issue #19567 · Graylog2/graylog2-server · GitHub

3

Hi,
I’ve checked the database and saw the ransom note READ__ME_TO_RECOVER_YOUR_DATA. Our MongoDB port was opened without --auth setup in the Docker container. Luckily, we have the backup data. We’ve set up credentials for GRAYLOG_MONGODB_URI and fixed our deployment.

Appreciate your help!

1 Like
4

Happy to help! Good that you have backups!

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.