Graylog leader wont run migrations after upgrade

1. Describe your incident:

upgraded graylog from 3.3. to 4.3. then upgraded the elastic cluster from 6.8 to 7.10.2.
leader node wont startup after elastic cluster upgrade, migrations keep failing

2. Describe your environment:

  • OS Information:
    ubuntu 20.04

  • Package Version:
    2x graylog nodes (one leader) - version: 4.3.7-1
    3x elastic nodes also running mongodb

  • Graylog Version: 4.3.7-1

  • Java Version: 11 on es nodes, 8 on graylog nodes

  • Elasticsearch Version: 7.10.2

  • MongoDB Version: 4.4

  • Operating System: 20.04

  • Browser version: chrome 105

  • Service logs, configurations, and environment variables:

2022-10-04T12:49:31.704-04:00 INFO  [ServerBootstrap] Graylog server 4.3.7+05bccc7 starting up
2022-10-04T12:49:31.705-04:00 INFO  [ServerBootstrap] JRE: Private Build 1.8.0_342 on Linux 5.15.0-1020-aws
2022-10-04T12:49:31.705-04:00 INFO  [ServerBootstrap] Deployment: deb
2022-10-04T12:49:31.705-04:00 INFO  [ServerBootstrap] OS: Ubuntu 20.04.5 LTS (focal)
2022-10-04T12:49:31.705-04:00 INFO  [ServerBootstrap] Arch: amd64
2022-10-04T12:49:31.819-04:00 INFO  [ServerBootstrap] Running 44 migrations...
2022-10-04T12:49:32.664-04:00 WARN  [RestClient] request [PUT http://ca04vlgraylogdb02.agilitypr.internal:9200/gl-events*,gl-system-events*/_mapping?master_timeout=30s&ignore_unavailable=true&expand_wildcards=open%2Cclosed&allow_no_indices=true&ignore_throttled=false&timeout=30s] returned 1 warnings: [299 Elasticsearch-6.8.23-4f67856 "[types removal] The parameter include_type_name should be explicitly specified in put mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', and requests are expected to omit the type name in mapping definitions."]
2022-10-04T12:49:32.753-04:00 WARN  [ServerBootstrap] Exception while running migrations
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: An error occurred:
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:151) ~[?:?]
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:111) ~[?:?]
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:104) ~[?:?]
        at org.graylog.storage.elasticsearch7.views.migrations.V20200730000000_AddGl2MessageIdFieldAliasForEventsES7.addGl2MessageIdFieldAlias(V20200730000000_AddGl2MessageIdFieldAliasForEventsES7.java:53) ~[?:?]
        at org.graylog.plugins.views.migrations.V20200730000000_AddGl2MessageIdFieldAliasForEvents.upgrade(V20200730000000_AddGl2MessageIdFieldAliasForEvents.java:76) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.lambda$runMigrations$0(ServerBootstrap.java:264) ~[graylog.jar:?]
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422) ~[graylog.jar:?]
        at com.google.common.collect.RegularImmutableSortedSet.forEach(RegularImmutableSortedSet.java:88) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.runMigrations(ServerBootstrap.java:261) ~[graylog.jar:?]
        at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:187) [graylog.jar:?]
        at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:311) [graylog.jar:?]
        at org.graylog2.bootstrap.Main.main(Main.java:45) [graylog.jar:?]
Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=action_request_validation_exception, reason=Validation Failed: 1: mapping type is missing;]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:187) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1565) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.putMapping(IndicesClient.java:353) ~[?:?]
        at org.graylog.storage.elasticsearch7.views.migrations.V20200730000000_AddGl2MessageIdFieldAliasForEventsES7.lambda$addGl2MessageIdFieldAlias$2(V20200730000000_AddGl2MessageIdFieldAliasForEventsES7.java:53) ~[?:?]
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:109) ~[?:?]
        ... 10 more
        Suppressed: org.graylog.shaded.elasticsearch7.org.elasticsearch.client.ResponseException: method [PUT], host [http://ca04vlgraylogdb02.agilitypr.internal:9200], URI [/gl-events*,gl-system-events*/_mapping?master_timeout=30s&ignore_unavailable=true&expand_wildcards=open%2Cclosed&allow_no_indices=true&ignore_throttled=false&timeout=30s], status line [HTTP/1.1 400 Bad Request]
Warnings: [[types removal] The parameter include_type_name should be explicitly specified in put mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', and requests are expected to omit the type name in mapping definitions.]
{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: mapping type is missing;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: mapping type is missing;"},"status":400}
                at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302) ~[?:?]
                at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:272) ~[?:?]
                at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestClient.performRequest(RestClient.java:246) ~[?:?]
                at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613) ~[?:?]
                at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598) ~[?:?]
                at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1565) ~[?:?]
                at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.IndicesClient.putMapping(IndicesClient.java:353) ~[?:?]
                at org.graylog.storage.elasticsearch7.views.migrations.V20200730000000_AddGl2MessageIdFieldAliasForEventsES7.lambda$addGl2MessageIdFieldAlias$2(V20200730000000_AddGl2MessageIdFieldAliasForEventsES7.java:53) ~[?:?]
                at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:109) ~[?:?]
                at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:104) ~[?:?]
                at org.graylog.storage.elasticsearch7.views.migrations.V20200730000000_AddGl2MessageIdFieldAliasForEventsES7.addGl2MessageIdFieldAlias(V20200730000000_AddGl2MessageIdFieldAliasForEventsES7.java:53) ~[?:?]
                at org.graylog.plugins.views.migrations.V20200730000000_AddGl2MessageIdFieldAliasForEvents.upgrade(V20200730000000_AddGl2MessageIdFieldAliasForEvents.java:76) ~[graylog.jar:?]
                at org.graylog2.bootstrap.ServerBootstrap.lambda$runMigrations$0(ServerBootstrap.java:264) ~[graylog.jar:?]
                at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422) ~[graylog.jar:?]
                at com.google.common.collect.RegularImmutableSortedSet.forEach(RegularImmutableSortedSet.java:88) ~[graylog.jar:?]
                at org.graylog2.bootstrap.ServerBootstrap.runMigrations(ServerBootstrap.java:261) ~[graylog.jar:?]
                at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:187) [graylog.jar:?]
                at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:311) [graylog.jar:?]
                at org.graylog2.bootstrap.Main.main(Main.java:45) [graylog.jar:?]

3. What steps have you already taken to try and solve the problem?
checked logs, tried service restarts

4. How can the community help?
any advice on help on how to fix this issue?

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

1 Like

closing, was mistake on my part, one es node service didnt get restarted correctly post upgrade (somehow!!??) resulting in a cluster with 2 es nodes on 7.10.2 and one still on 6.8. restarting the es service via

sudo service elasticsearch restart

resolved the issue

I used

curl -X GET "graylog01.test.internal:9200/"
curl -X GET "graylog02.test.internal:9200/"
curl -X GET "graylog03.test.internal:9200/"

to find the running es version of each host

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.