# JSON extraction in pipeline rules

**URL:** <https://community.graylog.org/t/json-extraction-in-pipeline-rules/2869>\
**Category:** Graylog Central (peer support)\
**Tags:** pipeline-rules\
**Created:** [October 23, 2017, 11:37am UTC](https://community.graylog.org/t/json-extraction-in-pipeline-rules/2869 "2017-10-23T11:37:48Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![mino](https://avatars.discourse-cdn.com/v4/letter/m/e68b1a/32.png) [@mino](https://community.graylog.org/u/mino)\
**Post date:** [October 23, 2017, 11:37am UTC](https://community.graylog.org/t/json-extraction-in-pipeline-rules/2869/1 "2017-10-23T11:37:48Z")

</div>

Hello everyone,

due to some extractor restrictions, I’m using pipelines to push log inputs from the Beats-Plugin into Graylog. The logs to be processed may contain a JSON object containing further informations like stacktraces, invoked methods and other informations. As those informations are optional for the logger, there are no defined keys for the JSON object to be defined. From this perspective, JSON data should be handled as arbitrary key-value data to be processed inside the pipeline.

Currently, my pipeline rule looks like:

```
rule "extract_json"
when
    has_field("json_data")
then
    let json = parse_json(to_string($message.json_data));
    let fields = select_jsonpath(json, {json_class: "$.class"});
    set_fields(fields);
    let fields = select_jsonpath(json, {json_method: "$.method"});
    set_fields(fields);
    let fields = select_jsonpath(json, {json_stacktrace: "$.stacktrace"});
    set_fields(fields);
end

```

Unfortunately, once a field to be extracted does not exists, the parser throws a NullPointer Exception and stops execution. Does anybody know a good option to parse arbitrary data inside a JSON object?

Many thanks in advance

---

<div class="post-metadata">

**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\
**Post date:** [October 23, 2017, 6:37pm UTC](https://community.graylog.org/t/json-extraction-in-pipeline-rules/2869/2 "2017-10-23T18:37:42Z")

</div>

> [@mino](#):
>
> Unfortunately, once a field to be extracted does not exists, the parser throws a NullPointer Exception and stops execution.

Please post the complete errors from the logs of your Graylog nodes.

---

<div class="post-metadata">

**Author:** ![mino](https://avatars.discourse-cdn.com/v4/letter/m/e68b1a/32.png) [@mino](https://community.graylog.org/u/mino)\
**Post date:** [October 24, 2017, 6:44am UTC](https://community.graylog.org/t/json-extraction-in-pipeline-rules/2869/3 "2017-10-24T06:44:56Z")

</div>

The error is not logged to the server log but as field **gl2\_processing\_error** in the message:

For rule ‘extract\_json’: In call to function ‘select\_jsonpath’ at 6:17 an exception was thrown: null

The message being published looks like:

```
2017-10-24T06:34:10.100 <component/server/I-7832> INFO: 'Testing filebeat and graylog' { "stacktrace": "some stacktrace to be extracted" , "method": "aMethodCalled" }

```

The extracted JSON (from a GROK pattern in the previous stage) is:

```
{ "stacktrace": "some stacktrace to be extracted" , "method": "aMethodCalled" }
```

---

<div class="post-metadata">

**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\
**Post date:** [October 24, 2017, 7:56am UTC](https://community.graylog.org/t/json-extraction-in-pipeline-rules/2869/4 "2017-10-24T07:56:17Z")

</div>

Please also check the logs of your Graylog node(s) for the corresponding error message.

---

<div class="post-metadata">

**Author:** ![mino](https://avatars.discourse-cdn.com/v4/letter/m/e68b1a/32.png) [@mino](https://community.graylog.org/u/mino)\
**Post date:** [October 24, 2017, 8:47am UTC](https://community.graylog.org/t/json-extraction-in-pipeline-rules/2869/5 "2017-10-24T08:47:04Z")

</div>

Unfortunately, I cannot find anything in the server.log file.

---

<div class="post-metadata">

**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\
**Post date:** [October 24, 2017, 11:34am UTC](https://community.graylog.org/t/json-extraction-in-pipeline-rules/2869/6 "2017-10-24T11:34:31Z")

</div>

I couldn’t reproduce the issue with the latest SNAPSHOT of Graylog using the rule and the example data you’ve provided.

If _guess_ it has been resolved with the following PR:

> <https://github.com/Graylog2/graylog-plugin-pipeline-processor/pull/210>

You can give Graylog 2.4.0-beta.1 a try and check if the issue has been resolved for you.

---

<div class="post-metadata">

**Author:** ![mino](https://avatars.discourse-cdn.com/v4/letter/m/e68b1a/32.png) [@mino](https://community.graylog.org/u/mino)\
**Post date:** [October 26, 2017, 7:38am UTC](https://community.graylog.org/t/json-extraction-in-pipeline-rules/2869/7 "2017-10-26T07:38:59Z")

</div>

Thanks jochen,

this perfectly looks like the issue I’m hitting.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [November 9, 2017, 7:39am UTC](https://community.graylog.org/t/json-extraction-in-pipeline-rules/2869/8 "2017-11-09T07:39:29Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.