ERROR [BlockingBatchedESOutput] ...ObjectNode to java.lang.String

Graylog Central
#1

Recently looked at the server logs for our graylog instances (v3.3.17) - ES (6.8.23)

Getting a lot (every few seconds) of the following messages below:

I guess my 1st question is, what is the impact of this and the 2nd would be, where to even start to decode/fix this?

Any help appreciated

2022-05-15T21:04:17.946+02:00 ERROR [BlockingBatchedESOutput] Unable to flush message buffer
java.lang.ClassCastException: Cannot cast com.fasterxml.jackson.databind.node.ObjectNode to java.lang.String
	at java.lang.Class.cast(Class.java:3369) ~[?:1.8.0_201]
	at org.graylog2.plugin.Message.getFieldAs(Message.java:643) ~[graylog.jar:?]
	at org.graylog2.plugin.Message.getMessage(Message.java:465) ~[graylog.jar:?]
	at org.graylog2.plugin.Message.toElasticSearchObject(Message.java:390) ~[graylog.jar:?]
	at org.graylog2.indexer.messages.Messages.bulkIndexChunk(Messages.java:268) ~[graylog.jar:?]
	at org.graylog2.indexer.messages.Messages.bulkIndexChunked(Messages.java:226) ~[graylog.jar:?]
	at org.graylog2.indexer.messages.Messages.bulkIndex(Messages.java:188) ~[graylog.jar:?]
	at org.graylog2.indexer.messages.Messages.bulkIndex(Messages.java:175) ~[graylog.jar:?]
	at org.graylog2.outputs.ElasticSearchOutput.writeMessageEntries(ElasticSearchOutput.java:111) ~[graylog.jar:?]
	at org.graylog2.outputs.BlockingBatchedESOutput.flush(BlockingBatchedESOutput.java:129) [graylog.jar:?]
	at org.graylog2.outputs.BlockingBatchedESOutput.forceFlushIfTimedout(BlockingBatchedESOutput.java:154) [graylog.jar:?]
	at org.graylog2.periodical.BatchedElasticSearchOutputFlushThread.doRun(BatchedElasticSearchOutputFlushThread.java:82) [graylog.jar:?]
	at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_201]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_201]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_201]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_201]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
#2

The message JSON contained an object where GL was expecting a string.
Recent GL versions have a separate stream for indexing errors, which is super helpful in trouble-shooting these things. Absent that, can you determine which messages are not being indexed?

#3

Thanks for the feedback… Interestingly this error actually causes all logs that were in the batch been sent to ES to be dropped completely. I have at least found the problem pod (which is forwarded via filebeat), removing this pop from logging stops the messages (and lost logs), however I am still unable to determine what is actually wrong with the logs… but the hunt continues.

#4

Post one of the raw log messages. What type of input / extractor / pipeline are you using to process the messages?

#5

After some trial and error the problematic pod was found, the dev contacted and corrected (they somehow were pushing an object and not a stringin the json (which then runs though some parsing pipelines in GL)… The annoying part is that no amount of debugging settings would indicate which log (or message) was the actual problem in the logs (graylog at least). Also the problem is not with indexing (so there are no errors around this) but purely pushing the logs to ES with the object (already indexed - i am speculating here) (which then drops the batch it seem).

However many thanks for the advise.

#6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.