Creating fields from query (pipelines?)

[mooseh (Louis Kidd)]

Hi Everyone

So i have been testing Graylog’s capabilities with parsing complex URL’s
what i’m trying to achieve is relatively simple but getting a graylog pipeline to play ball is
harder than i thought it would be.

so I have a field that is created called query which extracts all information after the ? using grok patterns in the url in the log, so for my query i have this:
test=true&animal=monkey

so for the pipeline I was thinking along these lines but i cannot seem to get it to be happy

rule "map query to fields"
when
   has_field("query")
then
    let data = parse_json(to_string($message.query));
    set_fields(to_map(data));
end

[macko003]

Your problem not fully clear for me, but.
You wrote a simple sting,and a pipeline, what try to process json.
Your sting not in json format.

I suggest something like that

Key=value extractor and quotes / remove whitespaces with regex and replace

a processing rule would look like: rule "kv_ex_prefix" when has_field("kv") then // extract all key-value from "message" and prefix it with kv_ set_fields( fields: key_value( value: to_string($message.message), trim_value_chars: "\"" ), prefix: "kv_" ); end This would also strip out the " if present.

[mooseh (Louis Kidd)]

yeah I knew the parse json was a longshot,

but what im basically trying to do is parse every single parameter given into the url
to be mapped into its own field

[Magneton (Jake Smith)]

Make sure that you have your logs in the correct format and then use a key value extractor using the & symbol to separate your fields on

Search the forums for several examples of using the key value extractor in a pipeline
Cheers

Magnetron

[benvanstaveren]

There is no real good way to do this with regular expressions (or json parsing since it’s not json :D) but the key/value extractor may do the trick. Alternatively, change the way your application logs, if possible, to do it in full/proper json, then it’s easy enough to pull out with the json parser.