Indexer failures after upgrade to ES 5.5

Graylog Central (peer support)
1

hi,

I upgraded Graylog to 2.3 and Elasticsearch to 5.5. Since that, there are indexer failures that are caused By field mappings:

{"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Can't parse [index] value [not_analyzed] for field [TimeSource], expected [true] or [false]"}}
{"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Can't parse [index] value [not_analyzed] for field [FilterId], expected [true] or [false]"}}
{"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Can't parse [index] value [not_analyzed] for field [ActiveProfile], expected [true] or [false]"}}

To me it seems these mappings are some default mappings, not created by me. When I read the field mappings for the index (graylog_1366), these fields no not even exist there. Where should I look at this?

2

hi,

this error message might be related:

  2017-08-10T09:00:09.678+03:00 WARN  [Indices] Couldn't create index graylog_1367. Error: {"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [String] declared on field [ThreadID]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [message]: No handler for type [String] declared on field [ThreadID]","caused_by":{"type":"mapper_parsing_exception","reason":"No handler for type [String] declared on field [ThreadID]"}}
    2017-08-10T09:00:09.679+03:00 ERROR [IndexRotationThread] Couldn't point deflector to a new index
    java.lang.RuntimeException: Could not create new target index <graylog_1367>.
            at org.graylog2.indexer.MongoIndexSet.cycle(MongoIndexSet.java:294) ~[graylog.jar:?]
            at org.graylog2.indexer.rotation.strategies.AbstractRotationStrategy.rotate(AbstractRotationStrategy.java:75) ~[graylog.jar:?]
            at org.graylog2.periodical.IndexRotationThread.checkForRotation(IndexRotationThread.java:113) ~[graylog.jar:?]
            at org.graylog2.periodical.IndexRotationThread.lambda$doRun$0(IndexRotationThread.java:77) ~[graylog.jar:?]
            at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_141]
            at org.graylog2.periodical.IndexRotationThread.doRun(IndexRotationThread.java:73) [graylog.jar:?]
            at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141]
            at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_141]
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_141]
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_141]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
            at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

I have not made my own custom mapping for this ThreadID, it comes from somewhere else (mongodb?)

3

I wonder if this is the same issue as earlier: https://github.com/Graylog2/graylog2-server/issues/2763
that happened when ES was updated.

4

tried deleting template graylog-internal and restarting graylog. Did not solve this.

5

I solved this by making a custom field mapping for field ThreadID. It seems that Elasticsearch has changed some field types with the major version update: https://www.elastic.co/guide/en/elasticsearch/reference/current/string.html

When updating, Graylog does not seem to switch the field type automatically to a supported? I wonder if issue https://github.com/Graylog2/graylog2-server/issues/4066 is also related (the beginning of that error message is missing)

6

Have you been using any custom index templates/index mappings in Elasticsearch before or did you just add them now?

7

I had about 20 custom index mappings before. Those fields were all OK. None of those mappings were of a string type. After I added the ThreadID mapping (type “keyword”) Graylog rotated index OK right away.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.