due to some extractor restrictions, I’m using pipelines to push log inputs from the Beats-Plugin into Graylog. The logs to be processed may contain a JSON object containing further informations like stacktraces, invoked methods and other informations. As those informations are optional for the logger, there are no defined keys for the JSON object to be defined. From this perspective, JSON data should be handled as arbitrary key-value data to be processed inside the pipeline.
Currently, my pipeline rule looks like:
rule "extract_json"
when
has_field("json_data")
then
let json = parse_json(to_string($message.json_data));
let fields = select_jsonpath(json, {json_class: "$.class"});
set_fields(fields);
let fields = select_jsonpath(json, {json_method: "$.method"});
set_fields(fields);
let fields = select_jsonpath(json, {json_stacktrace: "$.stacktrace"});
set_fields(fields);
end
Unfortunately, once a field to be extracted does not exists, the parser throws a NullPointer Exception and stops execution. Does anybody know a good option to parse arbitrary data inside a JSON object?