Graylog Stop Indexing

tdesaules · July 29, 2020, 10:17am

Graylog stop indexing event journal disk grow up

Hi there Graylog community

Like I says in the title I have an issue indexing event in Graylog. After few hours my craylog single node stop indexing event in the Elastisearch database.

Any idea that should help me ?

Issue overview

Output stay at “0” and my journal disk size grow up until 100%+ :

Graylog system version

Hostname:
graylog1
Node ID:
93578a01-da62-401c-8563-4147c029fe5e
Version:
3.3.2+ade4779, codename Sloth Rocket
JVM:
PID 16201, Oracle Corporation 1.8.0_252 on Linux 4.9.0-13-amd64
Time:
2020-07-29 11:50:25 +02:00

Elasticsearch Health

{
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 836,
  "active_shards" : 836,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

error I found

3 hours ago	graylog_2784	4012f260-d16f-11ea-809e-005056a775aa	{"type":"illegal_argument_exception","reason":"mapper [http_code] of different type, current_type [long], merged_type [keyword]"}

error log that I found from Elasticsearch :

[2020-07-29T11:58:24,543][INFO ][o.e.c.m.MetaDataMappingService] [graylog1-graylog] [graylog_2787/Tb9VBXgjSuCtg_rZWO_JVw] update_mapping [message]
[2020-07-29T11:58:24,561][INFO ][o.e.c.m.MetaDataMappingService] [graylog1-graylog] [graylog_2787/Tb9VBXgjSuCtg_rZWO_JVw] update_mapping [message]
[2020-07-29T11:58:24,584][INFO ][o.e.c.m.MetaDataMappingService] [graylog1-graylog] [graylog_2787/Tb9VBXgjSuCtg_rZWO_JVw] update_mapping [message]
[2020-07-29T11:58:24,611][INFO ][o.e.c.m.MetaDataMappingService] [graylog1-graylog] [graylog_2787/Tb9VBXgjSuCtg_rZWO_JVw] update_mapping [message]
[2020-07-29T11:58:24,654][INFO ][o.e.c.m.MetaDataMappingService] [graylog1-graylog] [graylog_2787/Tb9VBXgjSuCtg_rZWO_JVw] update_mapping [message]
[2020-07-29T11:58:24,669][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [graylog1-graylog] failed to put mappings on indices [[[graylog_2787/Tb9VBXgjSuCtg_rZWO_JVw]]], type [message]
java.lang.IllegalArgumentException: mapper [http_code] of different type, current_type [long], merged_type [keyword]
        at org.elasticsearch.index.mapper.FieldMapper.doMerge(FieldMapper.java:354) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.NumberFieldMapper.doMerge(NumberFieldMapper.java:1093) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:340) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:52) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.ObjectMapper.doMerge(ObjectMapper.java:487) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.RootObjectMapper.doMerge(RootObjectMapper.java:278) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.ObjectMapper.merge(ObjectMapper.java:457) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.RootObjectMapper.merge(RootObjectMapper.java:273) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.Mapping.merge(Mapping.java:91) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.DocumentMapper.merge(DocumentMapper.java:339) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:273) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:231) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:270) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:200) [elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:135) [elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) [elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-6.8.6.jar:6.8.6]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
[2020-07-29T11:58:24,670][DEBUG][o.e.a.b.TransportShardBulkAction] [graylog1-graylog] [graylog_2787][2] failed to execute bulk item (index) index {[graylog_deflector][message][0a42e010-d182-11ea-9fa3-005056a775aa], source[anonymization]}
java.lang.IllegalArgumentException: mapper [http_code] of different type, current_type [long], merged_type [keyword]
        at org.elasticsearch.index.mapper.FieldMapper.doMerge(FieldMapper.java:354) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.NumberFieldMapper.doMerge(NumberFieldMapper.java:1093) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:340) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:52) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.ObjectMapper.doMerge(ObjectMapper.java:487) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.RootObjectMapper.doMerge(RootObjectMapper.java:278) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.ObjectMapper.merge(ObjectMapper.java:457) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.RootObjectMapper.merge(RootObjectMapper.java:273) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.Mapping.merge(Mapping.java:91) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.index.mapper.DocumentMapper.merge(DocumentMapper.java:339) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:273) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:231) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:270) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:200) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:135) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-6.8.6.jar:6.8.6]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-6.8.6.jar:6.8.6]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
[2020-07-29T11:58:24,686][INFO ][o.e.c.m.MetaDataMappingService] [graylog1-graylog] [graylog_2787/Tb9VBXgjSuCtg_rZWO_JVw] update_mapping [message]

tdesaules · July 29, 2020, 1:01pm

Hi,

I think I just catch an error from Elasticsearch that is related :

2020-07-29T13:49:01.076+02:00 ERROR [SystemJobManager] Unhandled error while running SystemJob <6c9971c0-d191-11ea-a9d1-005056a775aa> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
java.lang.IllegalArgumentException: Cat response did not contain a JSON Array
        at io.searchbox.core.Cat.parseResponseBody(Cat.java:61) ~[graylog.jar:?]
        at io.searchbox.action.AbstractAction.createNewElasticSearchResult(AbstractAction.java:71) ~[graylog.jar:?]
        at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:44) ~[graylog.jar:?]
        at io.searchbox.core.Cat.createNewElasticSearchResult(Cat.java:16) ~[graylog.jar:?]
        at io.searchbox.client.http.JestHttpClient.deserializeResponse(JestHttpClient.java:212) ~[graylog.jar:?]
        at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:88) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:49) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:65) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.catIndices(Indices.java:581) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.getClosedIndices(Indices.java:520) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.Indices.isClosed(Indices.java:566) ~[graylog.jar:?]
        at org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob.execute(SetIndexReadOnlyAndCalculateRangeJob.java:66) ~[graylog.jar:?]
        at org.graylog2.system.jobs.SystemJobManager$1.run(SystemJobManager.java:89) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:241) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_252]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_252]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_252]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_252]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]

system · August 12, 2020, 1:01pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Connection to Elastic Stops Graylog Central (peer support)	7	675	February 22, 2018
Graylog elasticsearch -problem Graylog Central (peer support)	2	738	March 4, 2020
Index error after one ES node crashed Graylog Central (peer support)	7	1636	October 19, 2021
Problem with Graylog cluster Graylog Central (peer support)	4	546	December 15, 2020
Issue when loading system events Graylog Central (peer support)	5	203	May 30, 2023