Just installed graylog and getting slowly into it. I have forwarded many rsyslogs to my default stream.
No I have defined a search for e.g. “error” as a keyword for an event, but of course many messages containing the word “error” that are not worth triggering an alert. So I end up with a query that looks like
"error" AND NOT ("not connected to home DERP region" OR "routines::no shared cipher" OR "routines::wrong version number" OR "SSL_accept error from " OR "routines::bad key share" OR "routines::unsupported protocol" OR "lost connection")
which is basically : “select messages with keyword “error” but without several other strings that are characteristic to unwanted messages”.
And basically doing the same with “fail*”, “denied”, “unreachable” keywords.
Already now it is clear that this approach won’t scale, I have received 500+ alerts today which I still have to skim for waiving strings. Also the performance of the query will for sure degrade with more strings being excluded.
From what I’ve seen here, another approach would be to search for specific errors. But this implies that the query strings are already know. Is there a registry with samples available somewhere or how do you come to know what is relevant ?
Overall I assume my approach is not the recommended one. Seeking advice on how to proceed to kickstart my alerts setup
So i think you should use processing pipelines to extract or enrich your messages into individual fields.
Pipelines is a huge topic (but one of the most important things you can do in Graylog, there are lots of youtube videos and stuff though). At its most basic you could write some pipelines that have “when contains something AND contains somethinf else then set_field is error to yes” (thats not the format, just an explanation of what its doing). Or it could be a field of category = errorA errorB etc.
Once you have done that, writing alerts becomes trivial, your search can just be “category:errorA”.
many thanks for the prompt feedback.
besides the pipelines, I also found the solution of creating new Streams out of the default one, by selecting messages that match certain rules. With this, I can so to say move the messages I would else waive to a different stream, and with this keep all possibilities to create notifications in case certain type of entries come too often etc
This is what my current setup looks like
Now : how does this compare with your proposal with the pipelines ?
Furthermore : how to best write multiple selection conditions in the pipelines ? Are there known performance implications of using too many if/then conditions ?
I plan to monitor something like 40 containers and VMs, therfore not expecting too high number of messages/s
