# Extract nested json issue **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [February 1, 2024, 4:43pm UTC](https://community.graylog.org/t/extract-nested-json-issue/31390 "2024-02-01T16:43:31Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![rfinney](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rfinney/32/961_2.png) [@rfinney](https://community.graylog.org/u/rfinney)\ **Post date:** [February 1, 2024, 4:43pm UTC](https://community.graylog.org/t/extract-nested-json-issue/31390/1 "2024-02-01T16:43:31Z")
**1. Describe your incident:** I’m feeding DUO security logs into Graylog using Elastic Agent. I’m running an initial json extraction using the code I found from @tmacgbay. But when I try to run a second json extraction further down for the nested fields it’s not working. Data 1: ```auto {"access_device":{"epkey":"57KUNSYFRI2UYNRFW3","hostname":null,"ip":"10.10.10.10","location":{"city":"Atlanta","country":"United States","state":"Georgia"}},"alias":"","application":{"key":"57HD233375NY29","name":"Google Workspace - Admin and Staff"},"auth_device":{"ip":"11.11.11.11","key":"DPW35H33X23752397HN","location":{"city":"Atlanta","country":"United States","state":"Georgia"},"name":"000-000-0001"},"email":"user@company.com","event_type":"authentication","factor":"verified_duo_push","isotimestamp":"2024-02-01T16:08:57.496224+00:00","ood_software":null,"reason":"verification_code_correct","result":"success","timestamp":1706803737,"trusted_endpoint_status":"unknown","txid":"8ac223a3-035e-4033-a333-9337e33dc339","user":{"groups":["duo_it (from AD sync \"AD Sync Duo1\")","duo_pilot (from AD sync \"AD Sync Duo1\")"],"key":"DUOT63333HY9ZA233","name":"uname"}} ``` Pipeline Rule 1: ```auto rule "type duo-json-parser" when has_field("filebeat_input_type") AND (to_string($message.filebeat_input_type)) == "httpjson" then let the_json = parse_json(to_string($message.message)); //debug(concat("The json: ", to_string(the_json))); let the_map = to_map(the_json); //debug(concat("The map: ", to_string(the_map))); set_fields(the_map); end ``` This parses out everything except the nested data. In a pipeline further down I tried similar code to break out nested json in message fields access\_device, application, and auth\_device. Fields: ```auto access_device {"hostname":null,"epkey":"57KUNSYFRI2UYNRFW3","ip":"0.10.10.10","location":{"country":"United States","city":"Atlanta","state":"Georgia"}} application {"name":"Google Workspace - Admin and Staff","key":"57HD233375NY29"} auth_device {"ip":"11.11.11.11","name":"000-000-0001","location":{"country":"United States","city":"Atlanta","state":"Georgia"},"key":"DPW35H33X23752397HN"} ``` ```auto rule "type duo-json-parser-access_device" when has_field("access_device") then let the_json = parse_json(to_string($message.access_device)); debug(concat("The json: ", to_string(the_json))); let the_map = to_map(the_json); debug(concat("The map: ", to_string(the_map))); set_fields(the_map); end ``` But it doesn’t parse the data. I checked the debug logs and it’s showing: 2024-02-01T11:15:15.482-05:00 INFO [Function] PIPELINE DEBUG: The json: 2024-02-01T11:15:15.483-05:00 INFO [Function] PIPELINE DEBUG: The map: I confirmed it’s hitting the rule with a set\_field below the set\_fields set\_field(“xduo\_test”, “002”); **2. Describe your environment:** Graylog 5.2.3 CentOS Any ideas on how to accomplish this? Or why it’s not working? ---
**Author:** ![rfinney](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rfinney/32/961_2.png) [@rfinney](https://community.graylog.org/u/rfinney)\ **Post date:** [February 1, 2024, 5:52pm UTC](https://community.graylog.org/t/extract-nested-json-issue/31390/2 "2024-02-01T17:52:07Z")
Interestingly when I try to Add to query from the logs access\_device, application, or auth\_device it add this to the query: access\_device:“[object Object]” And when I look in the fields it doesn’t show access\_device, etc… ---
**Author:** ![rfinney](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/rfinney/32/961_2.png) [@rfinney](https://community.graylog.org/u/rfinney)\ **Post date:** [February 1, 2024, 6:24pm UTC](https://community.graylog.org/t/extract-nested-json-issue/31390/3 "2024-02-01T18:24:04Z")
So graylog doesn’t like nested json and it needs to be flat. > > > When using Packetbeat to send messages directly to Graylog2, the nested JSON obj…ects won't be decoded and would be seen as '\[object Object\],\[object Object\]' under search UI. Example input JSON message: > > \`\`\` > { > "\_index" : "graylog\_5", > "\_type" : "message", > "\_id" : "572b9193-16df-11e6-8a3b-000c2942c251", > "\_version" : 1, > "found" : true, > "\_source" : { > "packetbeat\_bytes\_in" : 32, > "packetbeat\_method" : "QUERY", > "packetbeat\_type" : "dns", > "packetbeat\_responsetime" : 140, > "packetbeat\_query" : "class IN, type A, conn.skype.com", > "gl2\_remote\_ip" : "172.16.220.1", > "packetbeat\_dns\_question\_name" : "conn.skype.com", > "gl2\_remote\_port" : 65532, > "packetbeat\_dns\_additionals\_count" : 0, > "packetbeat\_dns\_answers\_count" : 2, > "source" : "abs-MacBook-Pro.local", > "type" : "dns", > "gl2\_source\_input" : "572a39d0cdf3830902a406df", > "packetbeat\_dns\_response\_code" : "NOERROR", > "packetbeat\_direction" : "out", > "packetbeat\_client\_ip" : "192.168.0.3", > "packetbeat\_dns\_flags\_recursion\_allowed" : true, > "packetbeat\_dns\_flags\_truncated\_response" : false, > "packetbeat\_dns\_question\_class" : "IN", > "gl2\_source\_node" : "b6d4add1-2cfc-4fd1-b18d-0ad0478e00a8", > "packetbeat\_dns\_flags\_authoritative" : false, > "packetbeat\_status" : "OK", > "packetbeat\_client\_port" : 60426, > "timestamp" : "2016-05-10 18:45:16.558", > "packetbeat\_ip" : "192.168.0.1", > "packetbeat\_dns\_op\_code" : "QUERY", > "packetbeat\_bytes\_out" : 83, > "packetbeat\_dns\_flags\_recursion\_desired" : true, > "packetbeat\_transport" : "udp", > "packetbeat\_dns\_authorities\_count" : 0, > "packetbeat\_resource" : "conn.skype.com", > "streams" : \["572ae5c9cdf3830902a4bb7f" \], > "packetbeat\_dns\_answers" : \[ { > "class" : "IN", > "data" : "conn.skype.akadns.net", > "name" : "conn.skype.com", > "ttl" : 464, > "type" : "CNAME" > }, { > "class" : "IN", > "data" : "91.190.216.81", > "name" : "conn.skype.akadns.net", > "ttl" : 300, > "type" : "A" > } \], > "message" : "-", > "packetbeat\_dns\_question\_type" : "A", > "packetbeat\_count" : 1, > "name" : "MacBook-Pro.local", > "packetbeat\_dns\_id" : 62527, > "facility" : "packetbeat", > "packetbeat\_port" : 53 > } > } > \`\`\` > > packetbeat\_dns\_answers structure won't be decoded in this example. > > > Some messages may contain objects as field values and we need to properly conver…t them into strings that can be rendered by react. > > This commit changes the way we convert values into strings, relying on \`JSON.stringify()\` to do it. > > Refs Graylog2/graylog-plugin-beats#3. It may also be related to #2946. > > \*\*Update\*\*: As @joschi pointed out, this only fixes part of the problem in Graylog2/graylog-plugin-beats#3: being able to display messages with a hierarchical structure. The change does not affect how Graylog internally handles those messages. Code from @jivepig found here appears to have fixed the issue: > [@Parsing nested json message in field with parent object in pipeline](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/8?page=2): > > hey, Only thing I can think of is using regex and/or a lookup table. Within the pipeline you can use the lookup table/s. ```auto rule "Random User Data Flatten Json Rule" // From sample data : https://randomuser.me/api/ // Api input path: * when true then let sJson = to_string($message.message); let sJson = regex_replace( pattern: "^\\[|\\]$", value: sJson, replacement: "" ); let rsJson = flatten_json(to_string(sJson), "flatten"); set_fields(to_map(rsJson)); //remove_field("result"); //set_field("message", "parsed user data"); end ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [February 15, 2024, 6:24pm UTC](https://community.graylog.org/t/extract-nested-json-issue/31390/4 "2024-02-15T18:24:34Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.