Debugging Message Processing

Hi all,

I’ve started to receive warning messages in the graylog-server log file this morning.
The warning message I’m receiving is as below:

WARN [ProcessBufferProcessor] Unable to process message <dd1fc5d0-2610-11e9-b957-000c2941d251>: java.lang.ClassCastException: org.graylog.plugins.pipelineprocessor.ast.expressions.FunctionExpression cannot be cast to org.graylog.plugins.pipelineprocessor.ast.expressions.LogicalExpression

I refactored pretty much all of the pipelines that messages first get routed into when they get sent into graylog and all appeared to be working as expected, these warning messages seem to have started overnight.

Has anyone got any tips for debugging this without having to crawl through every pipeline and every rule to see if I can spot the issue?

AFAIK since the message failed to process, it isn’t stored in ES so, I can’t open the message up within the web UI to look at what it is/where it went to debug that way.

I did try to access some of the messages that are causing these warnings, just in case I could see them.

I had no luck with accessing them via http://<graylog-ip>:9000/messages/graylog_0/<message-id>

Any tips would be much appreciated.

you are right with the assumption that the messages are not stored.

We already have an issue about that and will improve the debbuging - but currently the only option is to check the rules/pipeline.

Maybe - as an idea - add one rule at the very beginning that writes a debug statement to the logfile and with that you might have an idea what input/source has the problems. You would need to include as much information as possible in the debug statement, like input id (gl2_source_input) the Remote IP seen by Graylog (gl2_remote_ip) - that is how I would start to look into this.

1 Like

Sounds like a good idea.

Going to take a break from it for a moment and grab a coffee, think some things over and then get back to it.

Just so I know I’m understanding you correctly, you’re suggesting to add a new rule to the beginning of the pipelines which will write a message to the server log so, in theory, when the failed processing error occurs, I should see the debug message just before it so, I know where to start looking, correct?

I’ll have to throw another stage in at the beginning of each pipeline I believe for this to work but, I’ll look at that after I’ve woken myself up a bit more.

Thanks!

Just so I know I’m understanding you correctly, you’re suggesting to add a new rule to the beginning of the pipelines which will write a message to the server log so, in theory, when the failed processing error occurs, I should see the debug message just before it so, I know where to start looking, correct?

that is my idea on this.

1 Like

Thanks for that, @jan

Added the debug message to the start of each pipeline and managed to track down where the issue was coming from. Thankfully, the first rule I clicked on to investigate was the right one!

Turns out, it was the old issue of having the following in the pipeline rule:

NOT to_string($message.field) == "value"

Resolve the processing error by changing the line to:

NOT ( to_string)($message.field) == "value" )

Warning appears to have stopped now, going to keep watching the log file just in case but, I believe that was the issue.

Additionally, should the level of that message not be Error instead of Warning?

all that can cause a message lost should be error yes - do you mind opening a feature request/bug report for that in the server repo?

1 Like

Issue has been opened on the graylog2-server repo.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.