Journal utilization is too high - elasticsearch running


(Alberto di Marco) #1

Hello everyone, I hope this won’t sound like the usual question answered by “read the docs”, but I’m at a loss right now.
As per subject line, my Graylog instance is currently reporting “Journal utilization is too high”, and indeed the journal contains almost 400K messages as of right now.
After googling and reading the docs, I have checked Elastic search (process is running, has been restarted, in the Node window in the WebUI, it says “Elasticsearch cluster is green. Shards: 16 active, 0 initializing, 0 relocating, 0 unassigned”), I have verified resource usage (disk space and speed, CPU, RAM,…) and everything appears to be fine, the machine isn’t doing much.
I’ve checked the elasticseach and graylog log files and all I managed to find is 160k+ instances of “failed to execute bulk item (index) BulkShardRequest [[graylog_3][2]] containing [40] requests org.elasticsearch.index.mapper.MapperParsingException: failed to parse [level]” which however appear to be parsing errors.
I’ve now stopped all inputs, to make sure the instance doesn’t explode, but the message count in the journal doesn’t go down at all.
Processing is running and the node is marked as ALIVE.

What I believe caused the issue, was that the disk filled up, meaning we had to stop everything, expand the disk (it’s a VM), reboot, restart Graylog, MongoDB and Elasticsearch and delete an older index.

Versions:
Graylog: 2.4.6+ceaa7e4
Elasticsearch: 5.6.7

What am I missing?


(Jan Doberstein) #2

Did you rotate the index? Go to System > Indices > YOURINDEX > Maintenance > Rotate Index

On the last creation the field level was created with a different type than before (you cut the error) and so it is not possible to ingest into elasticsearch because a conflict is given.


(Alberto di Marco) #3

Hi Jan,
I rotated the index now, a new index was created. Journal message count is unchanged after waiting about a minute.

I am actually unsure as to what might be causing those errors, since the code generating them has not changed in at least several weeks.

The whole error is:

[2018-10-08T07:16:40,520][DEBUG][o.e.a.b.TransportShardBulkAction] [7zNmUtL] [graylog_3][1] failed to execute bulk item (index) BulkShardRequest [[graylog_3][1]] containing [65] requests
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [level]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:298) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:468) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:591) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:396) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:373) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:66) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:277) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:530) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:507) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.bulk.TransportShardBulkAction.prepareIndexOperationOnPrimary(TransportShardBulkAction.java:458) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:466) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:146) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:115) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:70) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:975) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:944) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:345) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:270) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:924) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:921) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:151) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1659) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:933) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:92) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:291) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:266) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:248) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.7.jar:5.6.7]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
Caused by: java.lang.NumberFormatException: For input string: “INFO”
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043) ~[?:?]
at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110) ~[?:?]
at java.lang.Double.parseDouble(Double.java:538) ~[?:1.8.0_141]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:187) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$7.parse(NumberFieldMapper.java:737) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$7.parse(NumberFieldMapper.java:709) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.mapper.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:1072) ~[elasticsearch-5.6.7.jar:5.6.7]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:287) ~[elasticsearch-5.6.7.jar:5.6.7]
… 36 more


(Jan Doberstein) #4

what does the field level contains?


(Alberto di Marco) #5

it’s a string. Usually containing either “ERROR”, “WARN”,…


(Jan Doberstein) #6

You might want to check your Elasticsearch mapping. ( this part of the docs helps with that: http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#custom-index-mappings )


#7

You have two different problems at the same time.

  1. Elasticsearch mappings (see Jan’s post)
  2. Diskspace problems. Check your journal options and make sure you will never again go out of space. Then look at this: https://community.graylog.org/t/after-disk-space-issue-no-out-messages-help/1766

(system) #8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.