Pipeline Not Getting Throughput

[poisedforflight]

I am trying to implement the pipeline rules found here but it’s not processing any messages: Sonicwall Pipeline Rules - Templates and Rules Exchange / Pipeline Rules - Graylog Community

rule "Extract: Sonicwall Extraction"
when
    has_field("source") AND contains(to_string($message.source), "192.168.1.1", true)
then
    set_fields(
        fields:key_value(
            value:to_string($message.message),
            //remove double quotes from keys and values
            trim_value_chars:"\"", 
            trim_key_chars:"\""
            )
        );
end

This is on GL 5.0.

Anyone have any suggestions or is there any other info I can supply that would be helpful?

[faen]

Is the pipeline connected to the “Ridgeland SonicWall” stream?

In my application, I have my sonicwall only going into it’s own stream and not the Default Stream - not sure if that will make any difference or not, but might be worth a try if the pipeline is connected to the correct stream. Let me know if you want a screenshot of the pipeline connection, but it’s just below the “details” section. This is at the pipeline level and not the stage/rule level.

[poisedforflight]

It is:

[faen]

Thanks for the quick confirmation.

As next steps, I would try:

1. See if you can take the messages out of the default stream, so they are only in the sonicwall stream, and if that changes the behavior.
2. Ensure that you are looking at the most recent messages in search, and not an older message that may not have been processed (I know, obvious, but this was a stumbling block for me once, so I am throwing it out there)
3. Modify the rule to just write something for every message, so no “when” clause, just a set_field processed=true or something like that to try to isolate the error. Might be easier to make a new rule and move the rule in question to a later stage.

Hope that helps!

[poisedforflight]

@faen

set_field processed=true

1. I set to remove from default stream, no change
2. definitely looking at new messages
3. could you help me out with this. I’m not well versed in pipelines at all, trying to learn. What would the rule look like?

[faen]

Here’s a very simple rule that should result in a single additional field called “Processed”

rule "Test Rule"
When
    has_field("message")
Then
    set_field("Processed","true");
End

If you aren’t seeing those, I would focus on the input/stream area. Have you checked the graylog application log?