# Parsing nested json from JSON path HTTP API input **URL:** \ **Category:** Graylog Central (peer support)\ **Tags:** pipeline-rules, regex-special-charac\ **Created:** [January 13, 2022, 2:01pm UTC](https://community.graylog.org/t/parsing-nested-json-from-json-path-http-api-input/22323 "2022-01-13T14:01:04Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![jimbo](https://avatars.discourse-cdn.com/v4/letter/j/d78d45/32.png) [@jimbo](https://community.graylog.org/u/jimbo)\ **Post date:** [January 13, 2022, 2:01pm UTC](https://community.graylog.org/t/parsing-nested-json-from-json-path-http-api-input/22323/1 "2022-01-13T14:01:04Z")
Hi dear enthusiasts, I am using dockerised graylog 4.2 using docker-compose. I input json data from an api using JSON path HTTP API input. This is what the json data looks like from the api: ```auto { "processGroupStatus": { "id": "42c68e21-017d-1000-12f0-602b3f80ebdf", "name": "NiFi Flow", "statsLastRefreshed": "13:06:30 UTC", "aggregateSnapshot": { "id": "42c68e21-017d-1000-12f0-602b3f80ebdf", "name": "NiFi Flow", "connectionStatusSnapshots": [ { "id": "4cc9e230-017d-1000-d35e-d45ef3bb7970", "connectionStatusSnapshot": { "id": "4cc9e230-017d-1000-d35e-d45ef3bb7970", "groupId": "42c68e21-017d-1000-12f0-602b3f80ebdf", "name": "failure", "sourceName": "QueryRecord", "destinationName": "QueryRecord", "flowFilesIn": 0, "bytesIn": 0, "input": "0 (0 bytes)", "flowFilesOut": 0, "bytesOut": 0, "output": "0 (0 bytes)", "flowFilesQueued": 0, "bytesQueued": 0, "queued": "0 (0 bytes)", "queuedSize": "0 bytes", "queuedCount": "0", "percentUseCount": 0, "percentUseBytes": 0 }, "canRead": true }, { "id": "017d1005-d114-1cc9-a405-b613763081f4", "connectionStatusSnapshot": { "id": "017d1005-d114-1cc9-a405-b613763081f4", "groupId": "42c68e21-017d-1000-12f0-602b3f80ebdf", "name": "query", "sourceName": "QueryRecord", "destinationName": "Funnel", "flowFilesIn": 0, "bytesIn": 0, "input": "0 (0 bytes)", "flowFilesOut": 0, "bytesOut": 0, "output": "0 (0 bytes)", "flowFilesQueued": 0, "bytesQueued": 0, "queued": "0 (0 bytes)", "queuedSize": "0 bytes", "queuedCount": "0", "percentUseCount": 0, "percentUseBytes": 0 }, "canRead": true }, { "id": "017d1007-d114-1cc9-3afb-ea68c1dd1d71", "connectionStatusSnapshot": { "id": "017d1007-d114-1cc9-3afb-ea68c1dd1d71", "groupId": "42c68e21-017d-1000-12f0-602b3f80ebdf", "name": "failure", "sourceName": "QueryRecord", "destinationName": "Funnel", "flowFilesIn": 0, "bytesIn": 0, "input": "0 (0 bytes)", "flowFilesOut": 0, "bytesOut": 0, "output": "0 (0 bytes)", "flowFilesQueued": 0, "bytesQueued": 0, "queued": "0 (0 bytes)", "queuedSize": "0 bytes", "queuedCount": "0", "percentUseCount": 0, "percentUseBytes": 0 }, "canRead": true }, { "id": "4cc9c107-017d-1000-7e94-3d598af1d186", "connectionStatusSnapshot": { "id": "4cc9c107-017d-1000-7e94-3d598af1d186", "groupId": "42c68e21-017d-1000-12f0-602b3f80ebdf", "name": "success", "sourceName": "GenerateFlowFile", "destinationName": "QueryRecord", "flowFilesIn": 0, "bytesIn": 0, "input": "0 (0 bytes)", "flowFilesOut": 0, "bytesOut": 0, "output": "0 (0 bytes)", "flowFilesQueued": 0, "bytesQueued": 0, "queued": "0 (0 bytes)", "queuedSize": "0 bytes", "queuedCount": "0", "percentUseCount": 0, "percentUseBytes": 0 }, "canRead": true }, { "id": "017d1009-d114-1cc9-a097-8bcb313cc29f", "connectionStatusSnapshot": { "id": "017d1009-d114-1cc9-a097-8bcb313cc29f", "groupId": "42c68e21-017d-1000-12f0-602b3f80ebdf", "name": "query", "sourceName": "QueryRecord", "destinationName": "Funnel", "flowFilesIn": 0, "bytesIn": 0, "input": "0 (0 bytes)", "flowFilesOut": 0, "bytesOut": 0, "output": "0 (0 bytes)", "flowFilesQueued": 0, "bytesQueued": 0, "queued": "0 (0 bytes)", "queuedSize": "0 bytes", "queuedCount": "0", "percentUseCount": 0, "percentUseBytes": 0 }, "canRead": true } ], "processorStatusSnapshots": [ { "id": "4cc95d00-017d-1000-e5bc-3cf01768c2fd", "processorStatusSnapshot": { "id": "4cc95d00-017d-1000-e5bc-3cf01768c2fd", "groupId": "42c68e21-017d-1000-12f0-602b3f80ebdf", "name": "QueryRecord", "type": "QueryRecord", "runStatus": "Stopped", "executionNode": "ALL", "bytesRead": 0, "bytesWritten": 0, "read": "0 bytes", "written": "0 bytes", "flowFilesIn": 0, "bytesIn": 0, "input": "0 (0 bytes)", "flowFilesOut": 0, "bytesOut": 0, "output": "0 (0 bytes)", "taskCount": 0, "tasksDurationNanos": 0, "tasks": "0", "tasksDuration": "00:00:00.000", "activeThreadCount": 0, "terminatedThreadCount": 0 }, "canRead": true }, { "id": "4cc9b2b1-017d-1000-a554-56b39a4e1bb6", "processorStatusSnapshot": { "id": "4cc9b2b1-017d-1000-a554-56b39a4e1bb6", "groupId": "42c68e21-017d-1000-12f0-602b3f80ebdf", "name": "GenerateFlowFile", "type": "GenerateFlowFile", "runStatus": "Stopped", "executionNode": "ALL", "bytesRead": 0, "bytesWritten": 0, "read": "0 bytes", "written": "0 bytes", "flowFilesIn": 0, "bytesIn": 0, "input": "0 (0 bytes)", "flowFilesOut": 0, "bytesOut": 0, "output": "0 (0 bytes)", "taskCount": 0, "tasksDurationNanos": 0, "tasks": "0", "tasksDuration": "00:00:00.000", "activeThreadCount": 0, "terminatedThreadCount": 0 }, "canRead": true }, { "id": "017d1000-5d00-1cc9-d584-c33bd37f436b", "processorStatusSnapshot": { "id": "017d1000-5d00-1cc9-d584-c33bd37f436b", "groupId": "42c68e21-017d-1000-12f0-602b3f80ebdf", "name": "QueryRecord", "type": "QueryRecord", "runStatus": "Stopped", "executionNode": "ALL", "bytesRead": 0, "bytesWritten": 0, "read": "0 bytes", "written": "0 bytes", "flowFilesIn": 0, "bytesIn": 0, "input": "0 (0 bytes)", "flowFilesOut": 0, "bytesOut": 0, "output": "0 (0 bytes)", "taskCount": 0, "tasksDurationNanos": 0, "tasks": "0", "tasksDuration": "00:00:00.000", "activeThreadCount": 0, "terminatedThreadCount": 0 }, "canRead": true } ], "processGroupStatusSnapshots": [], "remoteProcessGroupStatusSnapshots": [], "inputPortStatusSnapshots": [], "outputPortStatusSnapshots": [], "flowFilesIn": 0, "bytesIn": 0, "input": "0 (0 bytes)", "flowFilesQueued": 0, "bytesQueued": 0, "queued": "0 (0 bytes)", "queuedCount": "0", "queuedSize": "0 bytes", "bytesRead": 0, "read": "0 bytes", "bytesWritten": 0, "written": "0 bytes", "flowFilesOut": 0, "bytesOut": 0, "output": "0 (0 bytes)", "flowFilesTransferred": 0, "bytesTransferred": 0, "transferred": "0 (0 bytes)", "bytesReceived": 0, "flowFilesReceived": 0, "received": "0 (0 bytes)", "bytesSent": 0, "flowFilesSent": 0, "sent": "0 (0 bytes)", "activeThreadCount": 0, "terminatedThreadCount": 0 } }, "canRead": true } ``` However, on graylog, it ends up looking like this after using a json extractor: ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/f/ff41ec29c3236d93473912b1aacb8de64d03af3b.png) As you can see, the non nested fields get parsed, but for some reason the nested fields get their colons converted to equal signs. I have already tried a number of different ways to address this using regex replace extractor, but it is not smart and I had to individually make an extractor for each nested section, which is not ideal and not very smart of me. I was wondering if anyone has any good methods using either extractors or pipelines to parse all the json message including the nested bits? Thanks you ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [January 14, 2022, 2:09am UTC](https://community.graylog.org/t/parsing-nested-json-from-json-path-http-api-input/22323/2 "2022-01-14T02:09:23Z")
Hello @jimbo && Welcome I’m looking into this but I have a couple questions. When you stated this… > [@jimbo](#): > > it ends up looking like this after using a json extractor: But then you stated this… > [@jimbo](#): > > using regex replace extractor, but it is not smart and I had to individually make an extractor for each nested section, I’m assuming on your JSON path HTTP API input you create a JSON extractor then you had to create other extractors to parse your field called “processGroupStatus\_aggregateSnapshot\_connectionStatusSnapshots” How did you configure that JSON extractor? that’s one long field. To be honest that looks like two fields put together. Some information I dug up , perhaps it will help. > [@JSON parsing in pipeline](https://community.graylog.org/t/json-parsing-in-pipeline/14729): > > Hi. Graylog 3.0.2+1686930 I’m trying to parse json fields in pipeline rule. In stage 0 I’m parsing “message” field, creating “Properties” field from it and parsing “Properties”: when has\_field("Properties") && contains(to\_string($message.message), "data", true) then let msg = parse\_json(to\_string($message.message)); let prop = select\_jsonpath(msg, {Properties: "$.Properties"}); set\_field("Properties", to\_string(prop.Properties)); let props = parse\_json(to\_string($… > [@JSON extraction in pipeline rules](https://community.graylog.org/t/json-extraction-in-pipeline-rules/2869): > > Hello everyone, due to some extractor restrictions, I’m using pipelines to push log inputs from the Beats-Plugin into Graylog. The logs to be processed may contain a JSON object containing further informations like stacktraces, invoked methods and other informations. As those informations are optional for the logger, there are no defined keys for the JSON object to be defined. From this perspective, JSON data should be handled as arbitrary key-value data to be processed inside the pipeline. Cu… > > > Sometimes users might want to parse and merge the JSON payload of a message > wit…h the Graylog message without knowing the complete structure of the payload > or without having a fixed structure which could be selectively merged by using > the \`json\_path()\` method. > > This commit essentially adds the possiblity to create a pipeline rule emulating > the existing JSON extractor: > > rule "json" > when > // some condition > then > let json = parse\_json(to\_string($message.some\_field)); > set\_fields(json); > end > > Refs: https://community.graylog.org/t/parse-unknown-json-with-pipelines/3293/7 ---
**Author:** ![jimbo](https://avatars.discourse-cdn.com/v4/letter/j/d78d45/32.png) [@jimbo](https://community.graylog.org/u/jimbo)\ **Post date:** [January 14, 2022, 11:33am UTC](https://community.graylog.org/t/parsing-nested-json-from-json-path-http-api-input/22323/3 "2022-01-14T11:33:31Z")
Hi there, Thank you @gsmith . > [@gsmith](#): > > I’m assuming on your JSON path HTTP API input you create a JSON extractor then you had to create other extractors to parse your field called “processGroupStatus\_aggregateSnapshot\_connectionStatusSnapshots” That is correct. I did create a JSON extractor with the following config: ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/e/e4f38702b6bb946a750d9894b74d1b7ea2751bbd.png) However, the resultant parsed data which I showed you with equal signs in the nested bits, were only a result of using the JSON extractor, Not any other extractors. That nested long field you are referring to is long and it is an array of JSON objects. Thank you for your help. I will have a look at those links. They seem rather helpful. 😃 ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [January 28, 2022, 11:33am UTC](https://community.graylog.org/t/parsing-nested-json-from-json-path-http-api-input/22323/4 "2022-01-28T11:33:59Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.