Issue with Key=Value Parser Pipeline for specific log-messages

Graylog Central (peer support)
,
1

Hi Guys

I’m relatively new to graylog and thanks to this community and the official documentation I was able to solve many of my previous issues. However this one bugs my mind since a few days and I can’t figure out where the problem is. The situation is as follows:
We have a Windows Server that writes quite unusually formatted logs. Graylog gets these logs via the Beats input. On the Windows Server we’ve installed sidecars (version 1.0.2). The logs are comming in and I’ve created some basic extractors that work fine. My problem however is the following part of the message, which I store to the field “Full_Response” via extractor:

"Email":"example@example.ex","Password":null,"PendingVerificationEmail":null,"FirstName":"Example","MiddleName":Example,"LastName":"Test123","BirthDate":"1999-01-01T00:00:00","Salutation":"XX","PhoneNumber":"123456789","DueLevel":1,"PendingLevel":0,"EmailVerified":true,"PhoneNumberVerified":true,"Nationality":"XX","Language":"de","Address":{"Country":"Wakanda","City":"Random","Street":"Randomstreet","StreetNumber":"111","PostCode":"1337"},"PendingVerification":null,"ID":"ID0000000","DocumentType":0,"DocumentNumber":null,"DocumentExpirationDate":null,"DocumentIssuingCountry":null,"IdentTimeStamp":null},"StatusCode":200,"ErrorMessage":null

It appears to be a “convenient” key=value format, however the values are separated by : and not =. So since I’ve found no way to change the way the “predefined” key=value converter, I stumbled upon Pipelines after a few google-searches. In my humble opinion this might be the way to glory for putting these key value=pairs to its respective fields. So I’ve created this Pipeline-Rule:

rule "key_value_parser"
when
    has_field("Full_Response")
then
    set_fields(
        fields:
            key_value(
                value: to_string($message.Full_Response),
                delimiters:",",
                kv_delimiters:":")
            );
end

This rule is added to a simple Pipeline (on stage 0) which is then connected to the correct stream. However no data gets stored in its own field and it seems the rule is not really executed. I’ve tried the debug() function, but somehow it didn’t log anything, so there really might be an issue on the Pipeline itself even before the rules is executed?
Also important to note: My sample message is stored in a custom field via extractor (as said before: Full_Response). So I checked the Message Processor Configuration and made sure that the Pipeline Processor runs after the Message Filter Chain as this is, in my understanding, needed that the extractors work before the pipline, although this might be a misinterpretation.

Some more information to the setup:

Long story short: Do you find an issue in my Pipeline rule trying to parse the message? Or is there even a simpler solution for my problem that I just was not able to find yet?

Many thanks for your inputs!

2

Try to use this, I’ve added parameters to remove "{} from key and values. But, it’s not perfect, because of Country contains json.

rule "key_value_parser"
when
    has_field("Full_Response")
then
    set_fields(
        fields:
            key_value(
                value: to_string($message.Full_Response),
                delimiters:",",
                kv_delimiters:":",
                trim_key_chars:"\"",
                trim_value_chars: "\"{}")
            );
end
1 Like
3

Hi shoothub

Thanks for your answer. You made me realise that a big chunk of the message actually is in JSON. So I was able to fix my issue with the JSON extractor, although it seems like your solution might have worked as well.

Thx again for the eye-opener. Case closed

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.