Post-upgrade version doesn't change

Graylog Central (peer support)
1

I have upgraded an old-ish graylog instance (3.0.2) running via docker-compose to the newest version I could find that did not require any changes or data migration (3.1.4).

The upgrade itself went fine - I removed the service, removed the old graylog image, pulled the graylog/graylog:3.1.4 image and started the service again.
Everything seems to work fine - however graylog itself displays the version as 3.0.2.
The original 3.0.2 doesn’t exist any on the host so I’m not even sure why its displaying this version.

This all began due to a different issue - where a service that runs once a week and sends ~5K messages per second for about 10-15 seconds wasn’t logged in graylog, I can see the messages incoming into graylog and being processed by the relevant input and stream, however none of the messages are saved (checked timestamps and thats not the issue).

Any help regarding the displayed version issue will be much appreciated.

Following is my compose file:

version: ‘2’
services:
mongodb:
restart: unless-stopped
image: mongo:3
environment:
- TZ=Asia/Jerusalem
volumes:
- mongo_data:/data/db
elasticsearch:
restart: unless-stopped
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.6.1
volumes:
- es_data:/usr/share/elasticsearch/data
environment:
- TZ=Asia/Jerusalem
- http.host=0.0.0.0
- transport.host=localhost
- network.host=0.0.0.0
- “ES_JAVA_OPTS=-Xms512m -Xmx512m”
ulimits:
memlock:
soft: -1
hard: -1
mem_limit: 1g
graylog:
restart: unless-stopped
image: graylog/graylog:3.1
volumes:
- graylog_journal:/usr/share/graylog
environment:
- TZ=Asia/Jerusalem
- GRAYLOG_PASSWORD_SECRET=
- GRAYLOG_ROOT_PASSWORD_SHA2=
- GRAYLOG_HTTP_EXTERNAL_URI=http://:9000/
links:
- mongodb:mongo
- elasticsearch
depends_on:
- mongodb
- elasticsearch
ports:
# Graylog web interface and REST API
- 9000:9000
# Syslog UDP
- 1514:1514/udp
- 5555:5555/udp
- 5556:5556/udp
volumes:
mongo_data:
driver: local
es_data:
driver: local
graylog_journal:
driver: local

2

he @alexmrhv

did you cleaned your browser cache after the update? Graylog is in parts an application in your browser and you might have leftovers in your cache from the old application and a full page reload or when you delete the browser cache will solve the problem.

Jan

1 Like
3

Hey @jan ,
Thanks for the swift reply, however I did try that, even tried from a PC that never accessed graylog - same behavior, still shows up as 3.0.2 at the bottom of the page.
Can this be something to do with the mongo/elastic instances not going down during the upgrade or the graylog using a local volume and still retains some of the information about itself?

4

he @alexmrhv

when you go to System > configuration what Graylog version does that show to you?

If that is still 3.0.2 you might have missed docker-compose up -d to make docker taking the new image.

5

Hey @jan
Couldn’t find the version under “System -> configuration” but did find it under “System -> Nodes”.
Sinde the “cluster” only has one node the version displayed there is:
" Version: 3.0.2+1686930, codename Space Moose "
Also I did not use the “-d” flag but did detach the process from the running shell via Ctrl+Z and then “bg” to unpause the process, looking at the “docker ps” output the container is running from the “graylog/graylog:3.1” image (Since I wrote the original post I tried to re-run it with the 3.1, 3.1.4 and 3.1.3 image tags, all show the same behavior).

6

Here is some amusing output from the “docker inspect” command on the running graylog container:

On one hand we have this:

        "Labels": {
            "com.docker.compose.config-hash": "7c2f6869211881e8140c1cca6e4032f30a142a4ecd638ed23400f82566ba2edb",
            "com.docker.compose.container-number": "1",
            "com.docker.compose.oneoff": "False",
            "com.docker.compose.project": "root",
            "com.docker.compose.service": "graylog",
            "com.docker.compose.version": "1.24.1",
            "com.microscaling.docker.dockerfile": "/Dockerfile",
            "com.microscaling.license": "Apache 2.0",
            "maintainer": "Graylog, Inc. <hello@graylog.com>",
            "org.label-schema.build-date": "",
            "org.label-schema.description": "Official Graylog Docker image",
            "org.label-schema.name": "Graylog Docker Image",
            "org.label-schema.schema-version": "1.0",
            "org.label-schema.url": "https://www.graylog.org/",
            "org.label-schema.vcs-ref": "248ec93cd91cfd836d8200457db351271eecba00",
            "org.label-schema.vcs-url": "https://github.com/Graylog2/graylog-docker",
            "org.label-schema.vendor": "Graylog, Inc.",
            "org.label-schema.version": "3.1.4"
        }

Along with this:

"Image": "graylog/graylog:3.1.4",

But then we have this:

        "Health": {
            "Status": "healthy",
            "FailingStreak": 0,
            "Log": [
                {
                    "Start": "2020-06-25T18:02:27.583633183+03:00",
                    "End": "2020-06-25T18:02:27.870575792+03:00",
                    "ExitCode": 0,
                    "Output": "{\"cluster_id\":\"77e932c8-6737-4a0f-96cb-3458f45de6b3\",\"node_id\":\"e90ea9ce-a271-4d64-a282-94459de8413f\",\"version\":\"3.0.2+1686930\",\"tagline\":\"Manage your logs in the dark and have lasers going and make it look like you're from space!\"}"
                },
                {
                    "Start": "2020-06-25T18:02:37.873243681+03:00",
                    "End": "2020-06-25T18:02:38.148865223+03:00",
                    "ExitCode": 0,
                    "Output": "{\"cluster_id\":\"77e932c8-6737-4a0f-96cb-3458f45de6b3\",\"node_id\":\"e90ea9ce-a271-4d64-a282-94459de8413f\",\"version\":\"3.0.2+1686930\",\"tagline\":\"Manage your logs in the dark and have lasers going and make it look like you're from space!\"}"
                },
                {
                    "Start": "2020-06-25T18:02:48.151100702+03:00",
                    "End": "2020-06-25T18:02:48.436626433+03:00",
                    "ExitCode": 0,
                    "Output": "{\"cluster_id\":\"77e932c8-6737-4a0f-96cb-3458f45de6b3\",\"node_id\":\"e90ea9ce-a271-4d64-a282-94459de8413f\",\"version\":\"3.0.2+1686930\",\"tagline\":\"Manage your logs in the dark and have lasers going and make it look like you're from space!\"}"
                },
                {
                    "Start": "2020-06-25T18:02:58.439176229+03:00",
                    "End": "2020-06-25T18:02:58.707128715+03:00",
                    "ExitCode": 0,
                    "Output": "{\"cluster_id\":\"77e932c8-6737-4a0f-96cb-3458f45de6b3\",\"node_id\":\"e90ea9ce-a271-4d64-a282-94459de8413f\",\"version\":\"3.0.2+1686930\",\"tagline\":\"Manage your logs in the dark and have lasers going and make it look like you're from space!\"}"
                },
                {
                    "Start": "2020-06-25T18:03:08.709738973+03:00",
                    "End": "2020-06-25T18:03:08.993291229+03:00",
                    "ExitCode": 0,
                    "Output": "{\"cluster_id\":\"77e932c8-6737-4a0f-96cb-3458f45de6b3\",\"node_id\":\"e90ea9ce-a271-4d64-a282-94459de8413f\",\"version\":\"3.0.2+1686930\",\"tagline\":\"Manage your logs in the dark and have lasers going and make it look like you're from space!\"}"
                }
            ]
        }

Notice how everything that’s not the graylog itself screams that this is 3.1.4 but graylog for some reason still thinks its 3.0.2…

7

Ok so the good news:
I managed to upgrade Graylog with all its components to the latest versions without losing any data.
(Went straight from 3.0.2 to 3.3.2 and it magically worked, sort of)

The bad news is that I kept the old setup and even after more tweaking the behavior is still the same.
Whats even funnier is that I used a clone of the same local volume for mongo and elastic for the new setup and all the versions are recognized as they should, so no clue why the graylog behavior previously discussed.

If this is somehow interesting for debugging/troubleshooting purposes I can provide whatever logs needed, otherwise this can be considered as closed.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.