Key_value in pipelines vs key_value on "input extractors"


#1

Hi,

we are trying to extract key values from a real world syslog message witch looks like this:

[zipflklatscher@4  webgate_Time="[13/Apr/2017:03:44:31 +0200]" username="XXXXX" client="xx.xxx.xx.xx" http_status="200" webgate_URL="https://dmp.xxxxx.com/d/607/2256/i/1.gif?c=xxxxxxxxxxxxxxx0&t=1&location=https%3A%2F%2Fwww.xxxxxx.de%2Fnews%2F2017%2F04%2F13%2F9-xxxx-in-xxx-vier-monaten-xxxx-schrowange-hat-abgespeckt.html&protocol=https%3A%2F%2F&tzo=-120&referrer=https%3A%2F%2Fwww.xxxxxxxxx.de%2F&date=1492060230&sw=1680&sh=1050&sd=24&pd=24&saw=1680&sah=1010&vw=1680&vh=904&pmk=Star%2C%20Promi%2C%20Stars%2C%20Promis%2C%20xxxxxh%2C%20Fotos%2C%20Bilder%2C%20xxxxxxx%2C%20News%2C%20Starnews%2C%20Starbilder%2C%20Videos&pmt=9%20Kilo%20in%20nur%20vier%20Monaten%3A%20Birgit%20Schrowange%20hat%20abgespeckt&csource=2&r=xxxxxxxxxxxxxxx4f7fa69" webgate_URLCategories="Business Zipfl" webgate_URLReputationString="Minimal Risk" mediatype="image/gif" out="561" in="1326" http_ua="Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko" webgate_AntimalwareVirusName="" webgate_ApplicationName="" webgate_BlockReason="" webgate_CacheStatus="TCP_MISS" protocol="HTTPS" webgate_Domain="xxxxsx.com" http_ref="https://www.xxxxxxxxxxxx.de/news/2017/04/13/9-xxxx-in-nur-vier-monaten-xxxxx-schrowange-hat-abgespeckt.html"  ]

by creating a “copy input” extractor based on simple key-value converter all fields seemd to be converted.

but by default key_value extractor used by a pipline, it does not the same as the extractor commented above.
By tweaking arround with the key_value arguments i am not able to set a delimiter by a regex combination, just by a list of strings

rule "webgate_key_value"
when
  has_field("application_name") && to_string($message.application_name) == "xxx"
then
    let msg = to_string($message.message);
    set_fields(key_value(value: msg
        ,delimiters: "\" "
        ,kv_delimiters: "="
        ,ignore_empty_values: true
        ,trim_key_chars: "\""
        ,trim_value_chars: "\""
        ));

end

how should the key_value looks like?

Thanks in advance
a.