Key=value from api url

Graylog Central (peer support)
1

I get a lot of search results for key=value problems when the value contains spaces, but our thing is the opposite:

we would like to extract key=value pairs from syslog messages like:

<158>1 2020-10-01T10:02:06.722989+02:00 server daemon 572 - [timeQuality tzKnown=“1” isSynced=“0”] Oct 01 10:02:06 daemon [5477]: 103.248.87.142 "POST /url/ActiveSync?User=username&DeviceId=H2QDT76P3PNAGKO&DeviceType=iPad&Cmd=Sync

Of course the tzKnown etc are extracted, but items like “User=username” and “DeviceType=iPad” are not extracted, I guess because there are not ‘ended’ by spaces, but are API urls, delimited by &

How can we make extract key=value work, also for those URL-like lines?

(Graylog 3.1.4)

2

Graylog can expand syslog structured data (tzKnown, isSynced) automatically, it’s not necessary to create KV extcrator for it. Edit Input and check Expand structured data? (Expand structured data elements by prefixing attributes with their SD-ID?)

To KV of values in URL:

  1. Create extractor (GROK/regex), to extract only URL parameters in field e.g activesyncurl
  2. Use pipeline rule which extract URL parameters, separated by &
rule "activesync url"
when
    has_field("activesyncurl")
then
    set_fields(key_value(
            value: to_string($message.activesyncurl),
            delimiters: "&",
            kv_delimiters: "=",
            ignore_empty_values: true,
            allow_dup_keys: true, // the default
            handle_dup_keys: ","  // meaning concat, default "take_first"
    ));
end
3

I enabled `expand structured data’ for the input (syslog UDP) in question, and removed the key=value extractor, but since that fields like tzKnown are no longer extracted…

Will need to take a closer look at stage two of your reply, with the pipeline rule etc. That’s new for me :slight_smile:

Thanks for the reply!

4

It should be extracted as fields: timeQuality_isSynced, timeQuality_syncAccuracy, timeQuality_tzKnown

5

Ah I missed that! Yes they are. Apologies!

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.