JSON extraction in pipeline rules


#1

Hello everyone,

due to some extractor restrictions, I’m using pipelines to push log inputs from the Beats-Plugin into Graylog. The logs to be processed may contain a JSON object containing further informations like stacktraces, invoked methods and other informations. As those informations are optional for the logger, there are no defined keys for the JSON object to be defined. From this perspective, JSON data should be handled as arbitrary key-value data to be processed inside the pipeline.

Currently, my pipeline rule looks like:

rule "extract_json"
when
    has_field("json_data")
then
    let json = parse_json(to_string($message.json_data));
    let fields = select_jsonpath(json, {json_class: "$.class"});
    set_fields(fields);
    let fields = select_jsonpath(json, {json_method: "$.method"});
    set_fields(fields);
    let fields = select_jsonpath(json, {json_stacktrace: "$.stacktrace"});
    set_fields(fields);
end

Unfortunately, once a field to be extracted does not exists, the parser throws a NullPointer Exception and stops execution. Does anybody know a good option to parse arbitrary data inside a JSON object?

Many thanks in advance


(Jochen) #2

Please post the complete errors from the logs of your Graylog nodes.


#3

The error is not logged to the server log but as field gl2_processing_error in the message:

For rule ‘extract_json’: In call to function ‘select_jsonpath’ at 6:17 an exception was thrown: null

The message being published looks like:

2017-10-24T06:34:10.100 <component/server/I-7832> INFO: 'Testing filebeat and graylog' { "stacktrace": "some stacktrace to be extracted" , "method": "aMethodCalled" }

The extracted JSON (from a GROK pattern in the previous stage) is:

{ "stacktrace": "some stacktrace to be extracted" , "method": "aMethodCalled" }

(Jochen) #4

Please also check the logs of your Graylog node(s) for the corresponding error message.


#5

Unfortunately, I cannot find anything in the server.log file.


(Jochen) #6

I couldn’t reproduce the issue with the latest SNAPSHOT of Graylog using the rule and the example data you’ve provided.

If guess it has been resolved with the following PR:

You can give Graylog 2.4.0-beta.1 a try and check if the issue has been resolved for you.


#7

Thanks jochen,

this perfectly looks like the issue I’m hitting.


(system) #8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.