Parse unknown JSON with pipelines

Graylog Central (peer support)
1

Hi everybody,

we write our Tomcat Log files in JSON by using log4j2 JSONLayout.
Each log file event is a single line with a whole JSON in it and the log file is parsed by using filebeat.
I know I can parse the JSON by using the JSON extractor from the Filter Chain, but I like to use the pipeline processor for it.
The number of JSON fields and values depends on the classes and is different in the log events.
Is it possible to parse an unknown JSON by using pipelines? I can only find possibilities to map known JSON keys and values.

Thanks and greetings,
ka51m1r

2

You can parse the JSON payload with parse_json() and then assign the result to the message with set_fields().

3

But then I have to know each field which I want to extract, right?

4

No, you don’t.

let json_fields = parse_json($message.custom_json_field);
set_fields(json_fields);
5

I have a whole JSON directly in the message field. Therefore I must parse the whole message.
When I use the function I get an error, that I a Map is expected but a JsonNode found.

let json_fields = parse_json(to_string($message));
set_fields(json_fields);

Do you have an idea why?

6

$message is an object which represents the currently processed message, not a specific field in that message.

If you want to access the field named “message”, you have to use $message.message.

To be honest, we’d better have named $message differently, e. g. $context or $ctx (similar to Elasticsearch’s Painless scripting), but now it’s too late for that…

7

Your right, but this was my fault.
Even if I declare $message.message, I always get an JsonNode and no Map and it is not possible to use the set_fields function.

let json_tree = parse_json(to_string($message.message));
set_fields(json_tree);

8

You’re correct, that was an oversight on my end.

I’ve created a PR to allow setting the parsed JSON payload using the set_fields() method:

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.