Pipeline silently dropping messages

[jpengenheiro]

Hi,

My graylog version is 2.4 today after a bit of troubleshooting I arrived at the following conclusion:

one of my pipelines is silently dropping messages after processing them

I have written a pipeline for parsing and enriching BGP messages I receive on my server,

the messages are first extracted for custom cisco fields, and afterwards matched in the pipeline if field cisco_mnemonic equals ADJCHANGE

this is the basic flow of it:

1. Lab device generates a neighbor shutdown message
2. tcpdump shows messages being received
3. debug statements for at least one rule in every stage of the pipeline shows up in the server log
4. the last stage is executed, with a route_to_stream statement to a dedicated stream
5. none of these messages show up in the web interface

The same behaviour is observed it the route_to_stream statement is absent,

The debug messages include fields I specifically set in the pipeline, so I assume my rules are doing what I expect ( I don’t have any drop_message statement )

1. the server logs do not show any processing error
2. only rules belonging to this pipeline appear to show this behaviour
3. there are no elasticsearch indexing errors

I have my server logging set to debug,

this is a pastebin log dump with a few examples

can anyone help me with understanding why this is happening?

EDIT

Upon further troubleshooting, I managed to find the cause of the problem

I was using the to_ip function inside a call to set_field, the former was the cause of the problem, after I replaced it with to_string things started working normally again,

I guess graylog didn’t like the field type being set to IP address and that’s probably my fault for trying, but still it’s at least strange that the whole 5 stage pipeline ran, and the problem was in the second stage

So the behaviour is still buggy IMO

[jtkarvo]

Try to debug so that you will not edit any message fields, but whenever you need to process a field, you save the result in a new field test. Then you’ll see, if the processing results are OK.

If you get messages through with that and the results seem to be OK, then you can try saving the processing results to the correct fields one by one and see which field is the problem.

The pastebin link did not work for me.

[jpengenheiro]

Hi,

the problem was, there was no way to see what was happening to the message as it was being processed, because it was silently dropped,

see my edit to the post to understand what I mean,

as soon as I called to_ip the whole message was lost, but somehow the rest of the pipeline still ran,

pastebin seems to be having some problems, if you are interested in seeing the logs I can provide them in a different platform