Hello. I’m very much new to Graylog and in need of a little expert advice.
I currently have a simple setup receiving a series of logs through a beat input. I’m applying three different extractors to them, each matching a specific keyword (it’s unique, so no log matches more than one of the extractors) and applying a Grok Pattern.
Since each of these matched patterns have slightly different fields, and very different message rates, I’d like to store them in separate indices (elastic) and control rotation individually.
So far, I’ve let them go to “All Messages” stream and setup a Pipeline where Stage 0 has three rules checking for the three keywords above. In each of them, I use route_to_stream to send the message to an individual stream (each keyword has a stream). Each of those streams is connected to a different index.
The routing seems to work, but the streams have duplicate messages. I made sure that “remove_from_default: true” is set and they are not showing up there. They just show up twice in each of the individual streams. I have Message Filter Chain before Pipeline Processor so that I can use the extracted fields from Input. Is this purely a visual issue, or am I actually storing these messages in two different indices (I’m assuming route_to_stream is removing the msg from the default index, no?)?
Clearly I’m doing something wrong, or missing a key bit of understanding.
I read that there’s been changes from Graylog 2.x to 3.x in regards to Pipelines vs Extractors, and a change in the Message Processor order. Since this is very much a learning experience for me, I’d really appreciate some guidance not only on how to make the above work, but also on what is the “best practice” approach to doing this in Graylog 3.x. Should I be applying the Grok patterns at the pipeline and forget the extractors?
Thanks!!