I want to write a pipeline rule to perform a search and report when any abnormal keywords are observed apart from regular or normal ones.
Example:-
Lets say I want to write a pipeline rule to detect suspicious user agents
Example:- if user agent contains contains below keywords its not suspicious :-
redbend
2)VN_SGW ,
3)r2h* ,
If user agent is different from above three keywords like if message contains :- python-requests, postman etc as the user agent then its suspicious.
I tried to write a rule like below but i think its wrong:-
rule "Suspicious user agents"
when (contains(to_string($message.message), "redbend", false) || contains(to_string($message.message), "VN_SGW", false) || contains(to_string($message.message), "r2h*", false))
then
route_to_stream(id:to_string("xxxxxxxxxxxxxxxxx"),remove_from_default:false);
end
Please help me to write a rule to perform a search when message does not contain specific keywords like above. I dont want to drop any message so drop_message() function doesn’t work in my case.
Hi @GrAlog_learner
I had to read it twice: you want to find everything, but the three keywords in your useragent. right?
Your rule looks quite good, but does the opposite: If it contains one of them, it will trigger. The section for Conditions in the docs will help you to fix that:
Expressions support the common boolean operators AND (or &&), OR (||), NOT (!), and comparison operators (<;, <=, >, >=,==, !=).
At the moment your query asks if one of the strings is there - you want to ask if non of them is there. My recomendation would be to put a negation in front of your condition:
rule "Suspicious user agents"
when NOT (contains(to_string($message.message), "redbend", false) || contains(to_string($message.message), "VN_SGW", false) || contains(to_string($message.message), "r2h*", false))
then
route_to_stream(id:to_string("xxxxxxxxxxxxxxxxx"),remove_from_default:false);
end
NOT (A OR B OR C) is the same as NOT A AND NOT B AND NOT C
Thanks for the response and i am sorry that you had to read it twice.
I am not very comfortable with pipeline rules and i am still learning the basics but i think now i understood how i can exclude certain keywords by keeping important keywords.
One more example for my understanding request you to verify the same :-
I want to write a rule where if my message contains x , y and z but it should not contain “a” and “b” so my pipeline rule would look something like this :-
rule "excluding unwanted keywords by keeping important keywords"
when
(contains(to_string($message.message), "x", true) || contains(to_string($message.message), "y", true) || contains(to_string($message.message), "z", true)) &&
NOT ((contains(to_string($message.message), "a", true)) || NOT (contains(to_string($message.message), "b", true)))
then
route_to_stream(id:to_string("xxxxxxxxxxxxxxxxx"),remove_from_default:false);
end
hi @GrAlog_learner
Why do you want to route all those messages to different streams? Do you need different data-retention? If yes, your way is the right one.
If not, my recommendation would be to have it filtered by a normal search and maybe use a Saved Seach?
We have around 12 plus customer environments using graylog. In each customer graylog environment we receive more than 100 GB’s of data everyday (purely k8’s logs and application component logs).
In each customer graylog environment we have a stream reserved for Security which contains logs which are relevant from security point of view.
In short we are sending only security related logs to a stream and with the help of syslog output whatever data is pushed to the stream gets forwarded to a SIEM - Centralized monitoring for all customers in a single place.
So my idea was to whitelist regular or unsuspicious keywords and cover only suspicious ones which are important from security point of view.
