I am new to Graylog. I have deployed the server and i am receiving messages from endpoints. I am working on filtering messages to extract only certain fields in all messages drop the rest. Below is my pipeline rule that i have applied to my default stream and i see througputs but it is not doing what it is expected to do.
rule “normalize_events”
when
has_field(“Category”)
then
remove_field(“Keywords”);
remove_field(“LogonTYpe”);
remove_field(“OpCode”);
remove_field(“OpCodeValue”);
remove_field(“ProcessID”);
remove_field(“ProviderGuid”);
remove_field(“RecordNumber”);
remove_field(“SeverityValue”);
remove_field(“SourceModuleName”);
remove_field(“SourceModuleType”);
remove_field(“TargetLogonId”);
remove_field(“TargetUserSid”);
remove_field(“Task”);
remove_field(“ThreadID”);
remove_field(“Version”);
remove_field(“Logon Type”);
remove_field(“level”);
end
For the moment, i am collecting logs from windows environment using NXLOG and output using GELF_UDP
Below is a sample event i would love to extract only few fields and drop the rest. Can someone point me in the right direction ?
Category
Logoff
Channel
Security
EventID
4634
EventReceivedTime
2023-08-22 15:35:45
EventType
AUDIT_SUCCESS
LogonType
3
Opcode
Info
OpcodeValue
0
Severity
INFO
SourceName
Microsoft-Windows-Security-Auditing
TargetDomainName
MYDOMNAIN
TargetUserName
Username
full_message
An account was logged off.
This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
message
An account was logged off.
@abamus, first, do you have a stream connected to the pipeline?
Second, do you see the fields you reference in the events as they come in?
Third, what is your Message Processors Configuration order in System/Configurations? Specifically, is the Message Filter Chain above or below the pipeline processor?
Yes i am running 5.1. Ive looked everywhere for this easy fix but i am not finding it. Also as i said earlier, i am new to Graylog but i ve successfully deployed it and ingesting my logs, created dashboards etc. The only part i am still struggling with is the pipeline tules
@abamus, am I correct to presume you are using an extractor to parse the messages originally? Do you mind sharing what kind?
I can’t see an obvious error, but it occurs to me that it may be in the hand-off between the extractor and the pipelline rule. It should work, since the message chain is running before the pipeline processor, but by moving them both into the pipeline, you could eliminate a step and use the simulator to test each rule.
If you are running 5.1 you can use the steps in this link to run the pipeline simulator and see if your rule is being matched, and what steps it is taking. Graylog 5.1 pipeline simulator | Arcade
So, you’re using a GELF input to collect them, then this pipeline to remove the fields?
@Joel_Duffield is right, the simulator may be a good place for you to track down what’s not working. The trace feature can give you some clues as it moves through the various stages.
The other thing you can do is to create a super simple rule. For example, one that says, when the condition matches, create a new field called test, with a value of “success”. If you can get that one working, try adding the remove field for one field and see if that works.
If every single message has the Category field, you can make the when condition a simple
