I upgraded Graylog to version 4.3 using yum on my Centos 7 server. However, after upgrade the server did not start and I found the following message from “/var/log/graylog-server/server.log”:
2022-05-30T20:14:57.326+03:00 INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2022-05-30T20:14:59.717+03:00 INFO [CmdLineTool] Loaded plugin: AWS plugins 4.3.0 [org.graylog.aws.AWSPlugin]
2022-05-30T20:14:59.718+03:00 INFO [CmdLineTool] Loaded plugin: Collector 4.3.0 [org.graylog.plugins.collector.CollectorPlugin]
2022-05-30T20:14:59.720+03:00 INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.3.0 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2022-05-30T20:14:59.720+03:00 INFO [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.3.0+7c09aad [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2022-05-30T20:14:59.720+03:00 INFO [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.3.0+7c09aad [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2022-05-30T20:14:59.820+03:00 INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=rpm
2022-05-30T20:15:01.689+03:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2022-05-30T20:15:01.850+03:00 INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-05-30T20:15:01.967+03:00 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:389}] to localhost:27017
2022-05-30T20:15:01.987+03:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 28]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=17597790}
2022-05-30T20:15:02.077+03:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:390}] to localhost:27017
2022-05-30T20:15:02.175+03:00 INFO [connection] Closed connection [connectionId{localValue:2, serverValue:390}] to localhost:27017 because the pool has been closed.
2022-05-30T20:15:02.178+03:00 INFO [MongoDBPreflightCheck] Connected to MongoDB version 4.0.28
2022-05-30T20:15:02.707+03:00 INFO [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <Elasticsearch:6.8.12>
2022-05-30T20:15:03.305+03:00 INFO [Version] HV000001: Hibernate Validator null
2022-05-30T20:15:13.178+03:00 INFO [InputBufferImpl] Message journal is enabled.
2022-05-30T20:15:13.265+03:00 INFO [NodeId] Node ID: 8eab9e8e-2e3e-4409-8ee5-06609758fe11
2022-05-30T20:15:13.998+03:00 INFO [LogManager] Loading logs.
2022-05-30T20:15:14.101+03:00 WARN [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000004392243507.index, deleting and rebuilding index...
2022-05-30T20:15:19.815+03:00 INFO [LogManager] Logs loading complete.
2022-05-30T20:15:19.821+03:00 INFO [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2022-05-30T20:15:19.845+03:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2022-05-30T20:15:19.859+03:00 INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-05-30T20:15:19.869+03:00 INFO [connection] Opened connection [connectionId{localValue:3, serverValue:391}] to localhost:27017
2022-05-30T20:15:19.871+03:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 28]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1095149}
2022-05-30T20:15:19.887+03:00 INFO [connection] Opened connection [connectionId{localValue:4, serverValue:392}] to localhost:27017
2022-05-30T20:15:20.303+03:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2022-05-30T20:15:21.511+03:00 INFO [ElasticsearchVersionProvider] Elasticsearch cluster is running Elasticsearch:6.8.12
2022-05-30T20:15:21.673+03:00 INFO [AbstractJestClient] Setting server pool to a list of 1 servers: [http://127.0.0.1:9200]
2022-05-30T20:15:21.674+03:00 INFO [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2022-05-30T20:15:21.931+03:00 INFO [JestClientFactory] Using custom ObjectMapper instance
2022-05-30T20:15:21.931+03:00 INFO [JestClientFactory] Node Discovery disabled...
2022-05-30T20:15:21.931+03:00 INFO [JestClientFactory] Idle connection reaping disabled...
2022-05-30T20:15:22.605+03:00 INFO [connection] Opened connection [connectionId{localValue:5, serverValue:393}] to localhost:27017
2022-05-30T20:15:23.734+03:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-05-30T20:15:24.865+03:00 INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-05-30T20:15:27.751+03:00 INFO [ServerBootstrap] Graylog server 4.3.0+7c09aad starting up
2022-05-30T20:15:27.752+03:00 INFO [ServerBootstrap] JRE: Red Hat, Inc. 1.8.0_332 on Linux 3.10.0-1127.19.1.el7.x86_64
2022-05-30T20:15:27.752+03:00 INFO [ServerBootstrap] Deployment: rpm
2022-05-30T20:15:27.752+03:00 INFO [ServerBootstrap] OS: CentOS Linux 7 (Core) (centos)
2022-05-30T20:15:27.752+03:00 INFO [ServerBootstrap] Arch: amd64
2022-05-30T20:15:28.347+03:00 INFO [ServerBootstrap] Running 44 migrations...
2022-05-30T20:15:30.891+03:00 WARN [ServerBootstrap] Exception while running migrations
java.lang.RuntimeException: Could not resolve new ref for condition on EventDefinition <5e2afd8f39b22502c984949e>. oldref <"161a79ec-6d31-4317-94db-5cb214b272aa"> refMap <{6a294191-44fa-4990-8a30-b01384324d9b=count-}>
at org.graylog2.migrations.V20200102140000_UnifyEventSeriesId.convertConditions(V20200102140000_UnifyEventSeriesId.java:116) ~[graylog.jar:?]
at org.graylog2.migrations.V20200102140000_UnifyEventSeriesId.lambda$convertConditions$1(V20200102140000_UnifyEventSeriesId.java:127) ~[graylog.jar:?]
at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_332]
at org.graylog2.migrations.V20200102140000_UnifyEventSeriesId.convertConditions(V20200102140000_UnifyEventSeriesId.java:124) ~[graylog.jar:?]
at org.graylog2.migrations.V20200102140000_UnifyEventSeriesId.lambda$convertConditions$1(V20200102140000_UnifyEventSeriesId.java:127) ~[graylog.jar:?]
at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_332]
at org.graylog2.migrations.V20200102140000_UnifyEventSeriesId.convertConditions(V20200102140000_UnifyEventSeriesId.java:124) ~[graylog.jar:?]
at org.graylog2.migrations.V20200102140000_UnifyEventSeriesId.unifySeriesId(V20200102140000_UnifyEventSeriesId.java:103) ~[graylog.jar:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_332]
at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_332]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_332]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_332]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_332]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_332]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_332]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_332]
at org.graylog2.migrations.V20200102140000_UnifyEventSeriesId.upgrade(V20200102140000_UnifyEventSeriesId.java:72) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.lambda$runMigrations$0(ServerBootstrap.java:263) ~[graylog.jar:?]
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422) ~[graylog.jar:?]
at com.google.common.collect.RegularImmutableSortedSet.forEach(RegularImmutableSortedSet.java:88) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.runMigrations(ServerBootstrap.java:261) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:187) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:311) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:45) [graylog.jar:?]
Is there a way to manually handle that ref for condition on EventDefinition, or a configuration option that lets you omit that check? How should I start fixing this?
Thanks in advance for any help you can offer!
