# Issue with Key=Value Parser Pipeline for specific log-messages **URL:** \ **Category:** Graylog Central (peer support)\ **Tags:** pipeline-rules, debuggingpl\ **Created:** [April 26, 2021, 10:30am UTC](https://community.graylog.org/t/issue-with-key-value-parser-pipeline-for-specific-log-messages/19606 "2021-04-26T10:30:42Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Eggman1337](https://avatars.discourse-cdn.com/v4/letter/e/9de053/32.png) [@Eggman1337](https://community.graylog.org/u/Eggman1337)\ **Post date:** [April 26, 2021, 10:30am UTC](https://community.graylog.org/t/issue-with-key-value-parser-pipeline-for-specific-log-messages/19606/1 "2021-04-26T10:30:42Z")
Hi Guys I’m relatively new to graylog and thanks to this community and the official documentation I was able to solve many of my previous issues. However this one bugs my mind since a few days and I can’t figure out where the problem is. The situation is as follows: We have a Windows Server that writes quite unusually formatted logs. Graylog gets these logs via the Beats input. On the Windows Server we’ve installed sidecars (version 1.0.2). The logs are comming in and I’ve created some basic extractors that work fine. My problem however is the following part of the message, which I store to the field “Full\_Response” via extractor: `"Email":"example@example.ex","Password":null,"PendingVerificationEmail":null,"FirstName":"Example","MiddleName":Example,"LastName":"Test123","BirthDate":"1999-01-01T00:00:00","Salutation":"XX","PhoneNumber":"123456789","DueLevel":1,"PendingLevel":0,"EmailVerified":true,"PhoneNumberVerified":true,"Nationality":"XX","Language":"de","Address":{"Country":"Wakanda","City":"Random","Street":"Randomstreet","StreetNumber":"111","PostCode":"1337"},"PendingVerification":null,"ID":"ID0000000","DocumentType":0,"DocumentNumber":null,"DocumentExpirationDate":null,"DocumentIssuingCountry":null,"IdentTimeStamp":null},"StatusCode":200,"ErrorMessage":null` It appears to be a “convenient” key=value format, however the values are separated by : and not =. So since I’ve found no way to change the way the “predefined” key=value converter, I stumbled upon Pipelines after a few google-searches. In my humble opinion this might be the way to glory for putting these key value=pairs to its respective fields. So I’ve created this Pipeline-Rule: ``` rule "key_value_parser" when has_field("Full_Response") then set_fields( fields: key_value( value: to_string($message.Full_Response), delimiters:",", kv_delimiters:":") ); end ``` This rule is added to a simple Pipeline (on stage 0) which is then connected to the correct stream. However no data gets stored in its own field and it seems the rule is not really executed. I’ve tried the debug() function, but somehow it didn’t log anything, so there really might be an issue on the Pipeline itself even before the rules is executed? Also important to note: My sample message is stored in a custom field via extractor (as said before: Full\_Response). So I checked the Message Processor Configuration and made sure that the Pipeline Processor runs after the Message Filter Chain as this is, in my understanding, needed that the extractors work before the pipline, although this might be a misinterpretation. Some more information to the setup: - Single server Graylog environment - Documentation used for Pipeline: [Functions — Graylog 4.0.0 documentation](https://docs.graylog.org/en/latest/pages/pipelines/functions.html#key-value) and this Blogpost: [Key Value Parser Delivers Useful Information Fast | Graylog](https://www.graylog.org/post/key-value-parser) - OS: Debian 10 (Buster) - Graylog Version: 4.0.5 Long story short: Do you find an issue in my Pipeline rule trying to parse the message? Or is there even a simpler solution for my problem that I just was not able to find yet? Many thanks for your inputs! ---
**Author:** ![shoothub](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/shoothub/32/6412_2.png) [@shoothub](https://community.graylog.org/u/shoothub)\ **Post date:** [April 26, 2021, 2:27pm UTC](https://community.graylog.org/t/issue-with-key-value-parser-pipeline-for-specific-log-messages/19606/2 "2021-04-26T14:27:55Z")
Try to use this, I’ve added parameters to remove "{} from key and values. But, it’s not perfect, because of Country contains json. ```auto rule "key_value_parser" when has_field("Full_Response") then set_fields( fields: key_value( value: to_string($message.Full_Response), delimiters:",", kv_delimiters:":", trim_key_chars:"\"", trim_value_chars: "\"{}") ); end ``` ---
**Author:** ![Eggman1337](https://avatars.discourse-cdn.com/v4/letter/e/9de053/32.png) [@Eggman1337](https://community.graylog.org/u/Eggman1337)\ **Post date:** [April 27, 2021, 8:39am UTC](https://community.graylog.org/t/issue-with-key-value-parser-pipeline-for-specific-log-messages/19606/3 "2021-04-27T08:39:29Z")
Hi shoothub Thanks for your answer. You made me realise that a big chunk of the message actually is in JSON. So I was able to fix my issue with the JSON extractor, although it seems like your solution might have worked as well. Thx again for the eye-opener. Case closed ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [May 11, 2021, 8:40am UTC](https://community.graylog.org/t/issue-with-key-value-parser-pipeline-for-specific-log-messages/19606/4 "2021-05-11T08:40:26Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.