Failed to migrate from Graylog Enterprise 5.2.4 to 6.0

Graylog Central (peer support)
,
1

Hello,

Graylog doesnt start after having upgraded its version from 5.2.4 to 6.0.1, with the error hereafter visible in the logs.

We are using Graylog Enterprise, running it on Ubuntu 20.04 focal with ElasticSearch 7.10.2 and MongoDB 7.0.9 (MongoDB was upgraded from 6.0.6 during installation of Graylog 6).

Any help would be appreciated.

2024-05-21T14:12:18.476Z WARN  [RestClient] request [PUT http://127.0.0.1:9200/gl-events_3/_mapping?master_timeout=30s&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false&ignore_throttled=false&timeout=30s] returned 1 warnings: [299 Elasticsearch-7.10.2-747e1cc71def077253878a59143c1f785afa92b9 "'y' year should be replaced with 'u'. Use 'y' for year-of-era. Use new java.time date format specifiers."]
2024-05-21T14:12:18.516Z ERROR [ServerBootstrap] Exception while running migrations
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: Unable to update index mapping gl-events_3
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:184) ~[?:?]
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:135) ~[?:?]
        at org.graylog.storage.elasticsearch7.IndicesAdapterES7.updateIndexMapping(IndicesAdapterES7.java:181) ~[?:?]
        at org.graylog.plugins.securityapp.migrations.V20240130141000_UpdateEventsIndexMapping.lambda$upgrade$0(V20240130141000_UpdateEventsIndexMapping.java:63) ~[?:?]
        at java.base/java.util.Optional.ifPresent(Unknown Source) ~[?:?]
        at org.graylog.plugins.securityapp.migrations.V20240130141000_UpdateEventsIndexMapping.lambda$upgrade$1(V20240130141000_UpdateEventsIndexMapping.java:56) ~[?:?]
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:423) ~[graylog.jar:?]
        at org.graylog.plugins.securityapp.migrations.V20240130141000_UpdateEventsIndexMapping.upgrade(V20240130141000_UpdateEventsIndexMapping.java:55) ~[?:?]
        at org.graylog2.bootstrap.ServerBootstrap.lambda$runMigrations$3(ServerBootstrap.java:392) ~[graylog.jar:?]
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source) ~[?:?]
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source) ~[?:?]
        at com.google.common.collect.CollectSpliterators$1WithCharacteristics.lambda$forEachRemaining$1(CollectSpliterators.java:72) ~[graylog.jar:?]
        at java.base/java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Unknown Source) ~[?:?]
        at com.google.common.collect.CollectSpliterators$1WithCharacteristics.forEachRemaining(CollectSpliterators.java:72) ~[graylog.jar:?]
        at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) ~[?:?]
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source) ~[?:?]
        at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
        at java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source) ~[?:?]
        at org.graylog2.bootstrap.ServerBootstrap.runMigrations(ServerBootstrap.java:389) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:315) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:335) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:267) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:55) [graylog.jar:?]
Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=illegal_argument_exception, reason=Error parsing [format] on field [timerange_end]: Invalid format: [uuuu-MM-dd HH:mm:ss.SSS]: Illegal pattern component: uuuu]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1565) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.putMapping(IndicesClient.java:353) ~[?:?]
        at org.graylog.storage.elasticsearch7.IndicesAdapterES7.lambda$updateIndexMapping$4(IndicesAdapterES7.java:181) ~[?:?]
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:133) ~[?:?]
        ... 23 more
2

You have run into a compatibility issue with ES6/ES7. I’m guessing the offending index was created with ES6? This causes ES7 to still apply legacy date format.

Looks like GL 6.0 uses some format strings that are not backwards compatible with ES6. That seems like a bug. I filed an issue.

An easy workaround is to ignore the migration failure by setting this in your graylog.conf:
ignore_migration_failures = true
The failing migration is specific to the Security product, which isn’t even available with ES. So you can safely ignore it.

3

Issue is fixed - will be included in the next bugfix release.

Meanwhile, setting ignore_migration_failures = true is your best workaround.

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.