# Graylog Pipeline rule **URL:** \ **Category:** Graylog Central (peer support)\ **Tags:** windows, nxlog\ **Created:** [August 22, 2023, 8:13pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892 "2023-08-22T20:13:59Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![abamus](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@abamus](https://community.graylog.org/u/abamus)\ **Post date:** [August 22, 2023, 8:13pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/1 "2023-08-22T20:13:59Z")
**1. Describe your incident:** Hello, I am new to Graylog. I have deployed the server and i am receiving messages from endpoints. I am working on filtering messages to extract only certain fields in all messages drop the rest. Below is my pipeline rule that i have applied to my default stream and i see througputs but it is not doing what it is expected to do. rule “normalize\_events” when has\_field(“Category”) then remove\_field(“Keywords”); remove\_field(“LogonTYpe”); remove\_field(“OpCode”); remove\_field(“OpCodeValue”); remove\_field(“ProcessID”); remove\_field(“ProviderGuid”); remove\_field(“RecordNumber”); remove\_field(“SeverityValue”); remove\_field(“SourceModuleName”); remove\_field(“SourceModuleType”); remove\_field(“TargetLogonId”); remove\_field(“TargetUserSid”); remove\_field(“Task”); remove\_field(“ThreadID”); remove\_field(“Version”); remove\_field(“Logon Type”); remove\_field(“level”); end For the moment, i am collecting logs from windows environment using NXLOG and output using GELF\_UDP Below is a sample event i would love to extract only few fields and drop the rest. Can someone point me in the right direction ? Category Logoff Channel Security EventID 4634 EventReceivedTime 2023-08-22 15:35:45 EventType AUDIT\_SUCCESS LogonType 3 Opcode Info OpcodeValue 0 Severity INFO SourceName Microsoft-Windows-Security-Auditing TargetDomainName MYDOMNAIN TargetUserName Username full\_message An account was logged off. Subject: Security ID: S-1-5-21-355331191-1240482024-1780769950-12996 Account Name: Account Domain: Logon ID: 0xB07E255 Logon Type: 3 This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer. message An account was logged off. Subject: Security ID: S-1-5-21- source timestamp 2023-08-22 19:35:45.000 ---
**Author:** ![Joel\_Duffield](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Joel\_Duffield](https://community.graylog.org/u/Joel_Duffield)\ **Post date:** [August 22, 2023, 8:54pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/2 "2023-08-22T20:54:27Z")
Have you confirmed that your pipeline rule is running at all, is it making changes to the measages, or are they being stored just as they come in? ---
**Author:** ![abamus](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@abamus](https://community.graylog.org/u/abamus)\ **Post date:** [August 23, 2023, 2:57am UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/3 "2023-08-23T02:57:34Z")
Yes my pipe line is running, it doesnt make any change to the messages that come in. They are stored as it is ---
**Author:** ![joe.gross](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/joe.gross/32/13372_2.png) [@joe.gross](https://community.graylog.org/u/joe.gross)\ **Post date:** [August 23, 2023, 6:28pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/4 "2023-08-23T18:28:31Z")
@abamus, first, do you have a stream connected to the pipeline? Second, do you see the fields you reference in the events as they come in? Third, what is your Message Processors Configuration order in System/Configurations? Specifically, is the Message Filter Chain above or below the pipeline processor? ---
**Author:** ![abamus](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@abamus](https://community.graylog.org/u/abamus)\ **Post date:** [August 23, 2023, 6:53pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/5 "2023-08-23T18:53:38Z")
@joe.gross yes my pipeline is connected to my default stream as shown below ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/b/bc5d64fd311a31dec112a0b6a16dbdfbda73ad45.png) Yes i still see the fields as the messages come in. Also, my Messge Filter Chain is above the pipeline processor ---
**Author:** ![abamus](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@abamus](https://community.graylog.org/u/abamus)\ **Post date:** [August 23, 2023, 6:54pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/6 "2023-08-23T18:54:09Z")
@joe.gross here is my Message Filter Chain as well ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/c/c113d31e2133da5954b8ca10b99bb2d535f5b932.png) ---
**Author:** ![joe.gross](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/joe.gross/32/13372_2.png) [@joe.gross](https://community.graylog.org/u/joe.gross)\ **Post date:** [August 23, 2023, 9:16pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/7 "2023-08-23T21:16:30Z")
Hmm… Can you post a screenshot of one message expanded to show the parsed fields please? Please choose one that has the field “Category”. ---
**Author:** ![abamus](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@abamus](https://community.graylog.org/u/abamus)\ **Post date:** [August 24, 2023, 12:34pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/8 "2023-08-24T12:34:10Z")
@joe.gross Below is a messaged with field ‘Category’ ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/3/338328c8fe60dee36ed123c310a6261cacb6e124.png) ---
**Author:** ![faen](https://avatars.discourse-cdn.com/v4/letter/f/71e660/32.png) [@faen](https://community.graylog.org/u/faen)\ **Post date:** [August 24, 2023, 2:36pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/9 "2023-08-24T14:36:55Z")
Is the GELF\_UDP input part of the default stream, or is it in a different stream? ---
**Author:** ![abamus](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@abamus](https://community.graylog.org/u/abamus)\ **Post date:** [August 24, 2023, 2:54pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/10 "2023-08-24T14:54:59Z")
@faen yes it is part of the default stream, that is my global inputs ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/3/3e8516a0748425f9ef65d443e83fb12454618b09.png) ---
**Author:** ![Joel\_Duffield](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Joel\_Duffield](https://community.graylog.org/u/Joel_Duffield)\ **Post date:** [August 25, 2023, 1:58am UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/11 "2023-08-25T01:58:35Z")
Are you running Graylog 5.1.x? If so there is a somewhat easy way to troubleshoot this. ---
**Author:** ![abamus](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@abamus](https://community.graylog.org/u/abamus)\ **Post date:** [August 25, 2023, 2:11am UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/12 "2023-08-25T02:11:17Z")
Yes i am running 5.1. Ive looked everywhere for this easy fix but i am not finding it. Also as i said earlier, i am new to Graylog but i ve successfully deployed it and ingesting my logs, created dashboards etc. The only part i am still struggling with is the pipeline tules ---
**Author:** ![joe.gross](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/joe.gross/32/13372_2.png) [@joe.gross](https://community.graylog.org/u/joe.gross)\ **Post date:** [August 28, 2023, 9:02pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/13 "2023-08-28T21:02:27Z")
@abamus, am I correct to presume you are using an extractor to parse the messages originally? Do you mind sharing what kind? I can’t see an obvious error, but it occurs to me that it may be in the hand-off between the extractor and the pipelline rule. It should work, since the message chain is running before the pipeline processor, but by moving them both into the pipeline, you could eliminate a step and use the simulator to test each rule. ---
**Author:** ![abamus](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@abamus](https://community.graylog.org/u/abamus)\ **Post date:** [August 28, 2023, 10:46pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/14 "2023-08-28T22:46:54Z")
@joe.gross i am not using extractor to parse the message. I am running the pipeline rule directly against the incoming messages. ---
**Author:** ![joe.gross](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/joe.gross/32/13372_2.png) [@joe.gross](https://community.graylog.org/u/joe.gross)\ **Post date:** [August 29, 2023, 8:26pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/15 "2023-08-29T20:26:11Z")
Ah. OK. Good. Glad I asked. Please share your pipeline details. It should look something like this: ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/c/c200944539ba3cadfdd9462db9d758ace3b57ca0.png) ---
**Author:** ![abamus](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@abamus](https://community.graylog.org/u/abamus)\ **Post date:** [August 29, 2023, 8:51pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/17 "2023-08-29T20:51:55Z")
@joe.gross ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/6/6c9625319c36e7e9a29e22862390e8023a059b3e.png) ---
**Author:** ![Joel\_Duffield](https://avatars.discourse-cdn.com/v4/letter/j/71c47a/32.png) [@Joel\_Duffield](https://community.graylog.org/u/Joel_Duffield)\ **Post date:** [September 12, 2023, 12:16am UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/18 "2023-09-12T00:16:29Z")
If you are running 5.1 you can use the steps in this link to run the pipeline simulator and see if your rule is being matched, and what steps it is taking. [Graylog 5.1 pipeline simulator | Arcade](https://app.arcade.software/share/0yQy0lkM072CoNzk1ZOz) ---
**Author:** ![joe.gross](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/joe.gross/32/13372_2.png) [@joe.gross](https://community.graylog.org/u/joe.gross)\ **Post date:** [September 12, 2023, 3:30pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/19 "2023-09-12T15:30:10Z")
So, you’re using a GELF input to collect them, then this pipeline to remove the fields? @Joel_Duffield is right, the simulator may be a good place for you to track down what’s not working. The trace feature can give you some clues as it moves through the various stages. The other thing you can do is to create a super simple rule. For example, one that says, when the condition matches, create a new field called test, with a value of “success”. If you can get that one working, try adding the remove field for one field and see if that works. If every single message has the Category field, you can make the when condition a simple when true then do\_stuff end. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [September 26, 2023, 3:31pm UTC](https://community.graylog.org/t/graylog-pipeline-rule/29892/20 "2023-09-26T15:31:03Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.