The initial goal seemed trivial, use a pipeline to enrich my IDS streams with Threat Intel data. Unfortunately nothing I have tried has worked thus far.
First issue seems to be that if you select “remove matches from all messages” for a stream and then use that stream in a pipeline - the pipeline always registers 0 msgs/sec. In order to see messages in the pipeline, I have to leave the messages in All Messages and connect the pipeline to the All Messages stream instead.
The input(s) for this data are JSON feeds from Snort and Bro and I am using JSON extractors on the input to parse the data into useful fields.
However, no matter what I try (and I’ve following several google rabbitholes on this) none of my rule syntax seems to trigger - not even a simple has_field test rule. I’m thinking this is because the raw message is JSON, but I’m not sure.
This is happening on both Graylog 2.4.x and 3.x