Upgrade to 4.3 error

[maxplay]

Yesterday I tried to upgrade from 4.2.9 to 4.3 but the graylog-server gives an error an won’t start.

My environment:

• OS Information: Ubuntu 18.04

• Package Version: graylog 4.3

The file /var/log/graylog-server/server.log shows this error:

2022-05-26T10:52:54.784-03:00 WARN  [ServerBootstrap] Exception while running migrations
java.lang.IllegalStateException: Index prefix conflict: a non-events index-set with prefix <gl-events> already exists. Configure a different <default_events_index_prefix> value in the server config file.
        at org.graylog2.migrations.V20190705071400_AddEventIndexSetsMigration.checkIndexPrefixConflicts(V20190705071400_AddEventIndexSetsMigration.java:131) ~[graylog.jar:?]
        at org.graylog2.migrations.V20190705071400_AddEventIndexSetsMigration.ensureEventsStreamAndIndexSet(V20190705071400_AddEventIndexSetsMigration.java:112) ~[graylog.jar:?]
        at org.graylog2.migrations.V20190705071400_AddEventIndexSetsMigration.upgrade(V20190705071400_AddEventIndexSetsMigration.java:85) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.lambda$runMigrations$0(ServerBootstrap.java:263) ~[graylog.jar:?]
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422) ~[graylog.jar:?]
        at com.google.common.collect.RegularImmutableSortedSet.forEach(RegularImmutableSortedSet.java:88) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.runMigrations(ServerBootstrap.java:261) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:187) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:311) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:45) [graylog.jar:?]

What I tried:

• Reviewed the server.conf file, changed the word “master” to “leader”, no luck.
• Tried to downgrade, no luck.
• Added this to the config file:

# Configures the prefix used for graylog event indices
# Default: gl-events
default_events_index_prefix = gl-events

→ no luck
should I define a new index prefix?

As I understand from the error message the upgrade conflicts with the existing prefix index…¿?

@jan could you take a quick-look? thanks!

I want any tip that can help me revert the upgrade or make this release 4.3 work

Thank you very much!
Max

[waab76 (Rob Curtis)]

Graylog is trying to create a new index set for Graylog server events with the index prefix gl-events and failing because an index set already exists in your setup with that prefix. According to the error message, you should update your config file to set default_events_index_prefix to anything other than gl-events.

[maxplay]

@waab76

default_events_index_prefix

well, your suggestion was what I was thinking of; although for an upgrade I don’t think that’s the ‘cleanest’ solution.

I added to my server.conf file:

# Configures the prefix used for graylog event indices
# Default: gl-events
# 26May22 - CHANGED due to upgrade to 4.3
default_events_index_prefix = gl-evts

# Configures the prefix used for graylog system event indices Default: gl-system-events
# 26May22 - CHANGED due to upgrade to 4.3
default_system_events_index_prefix = gl-system-evts

Also changed the line:

is_master = true

to

is_leader = true

Now is working.

Thank you very much!

Max

[patrickmann]

@maxplay

although for an upgrade I don’t think that’s the ‘cleanest’ solution.

This is not the regular upgrade behavior. The error is raised by a check designed to prevent problems when an ES instance is non-standard (manually modified, used by multiple applications, etc.).
It would be interesting to look at the existing indexes that already use the gl-events prefix. Where do they come from? If they are from an older GL version, what is the index template type (expected to be events)?

On a side note, what prompted you to change is_master to is_leader?

[maxplay]

@patrickmann

This is not the regular upgrade behavior. The error is raised by a check designed to prevent problems when an ES instance is non-standard (manually modified, used by multiple applications, etc.).
It would be interesting to look at the existing indexes that already use the gl-events prefix. Where do they come from? If they are from an older GL version, what is the index template type (expected to be events)?

Well, I don’t have the enough knowledge to answer these questions, maybe another person; maybe someone from the graylog support staff @waab76 ?

@patrickmann I changed that line in config file because of changes in 4.3.X version.

You can read https://docs.graylog.org/docs/server-conf for more information on server.conf config file.

Or read: https://docs.graylog.org/docs/upgrading-to-graylog-43x
for changes in 4.3.X version

Regards,
Max