# Parsing nested json message in field with parent object in pipeline

**URL:** <https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292>\
**Category:** Graylog Central (peer support)\
**Tags:** pipeline-rules\
**Created:** [October 25, 2022, 10:39pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292 "2022-10-25T22:39:32Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![brijesh.kalavadia](https://avatars.discourse-cdn.com/v4/letter/b/a6a055/32.png) [@brijesh.kalavadia](https://community.graylog.org/u/brijesh.kalavadia)\
**Post date:** [October 25, 2022, 10:39pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/1 "2022-10-25T22:39:32Z")

</div>

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.  
**Don’t forget to select tags to help index your topic!**

**1. Describe your incident:**  
I have set up pipeline to extract nested json and parse message to fields … it seems parsing works fine but we want something like prefixing its object with field name…  
for example I have below json:  
`{"@timestamp": "2022-10-25T17:55:29+00:00", "source": "mr-hub-nginx", "nginx": {"remote_addr": "xx.xx.101.216, xx.xx.146.165", "remote_user": "39942", "body_bytes_sent": 0, "request_length": 786, "request_time": 0.514, "status": 202, "request": "PATCH /test/v1/enablement/19299 HTTP/1.0", "request_method": "PATCH", "http_origin": "-", "http_referrer": "-", "site": "mr-hub-kube.test.com", "port": 443, "http_user_agent": "python-requests/2.28.1" }}`

We got logs with filed name “nginx,port,request,…” however, we want to have field name prefix by its object “nginx” like “nginx\_port, nginx\_request”

In short, whatever object we have should pick dynamically and prefix to field… is that possible ?

**2. Describe your environment:**

- OS Information: Ubuntu 20

- Package Version: 4.3.3+86369d3, codename _Noir_

- Service logs, configurations, and environment variables:  
I have pipeline set up:  
Stage0: extract json

```auto
rule "extract json"
when 
    regex("(\\{.*\\})", to_string($message.message)).matches == true
then
    let json = regex("(\\{.*\\})", to_string($message.message), ["json"])["json"];
    set_field("json", json);
end

```

Stage1: parse json

```auto
rule "parse json"
when
  has_field("json")
then
  // the following date format assumes there's no time zone in the string
 let json_props = parse_json(to_string($message.json));
 set_fields(to_map(json_props));
 
 let nginx_json = select_jsonpath(json_props, {nginx: "$.nginx"});
 let nginx_props = parse_json(to_string(nginx_json.nginx));
 set_fields(to_map(nginx_props));
end

```

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [October 25, 2022, 11:27pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/2 "2022-10-25T23:27:39Z")

</div>

Hello @brijesh.kalavadia

Correct me if I’m wrong but you want to rename the fields that have been created already? If so have you try creating **Stage 2** new rule and using `set_fields` for the naming convention?

---

<div class="post-metadata">

**Author:** ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)\
**Post date:** [October 26, 2022, 2:59pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/3 "2022-10-26T14:59:07Z")

</div>

the `set_fields()` function allows you to set the prefix for all fields it is working on:

[https://docs.graylog.org/docs/functions-1#set\_fields](https://docs.graylog.org/docs/functions-1#set_fields)

---

<div class="post-metadata">

**Author:** ![brijesh.kalavadia](https://avatars.discourse-cdn.com/v4/letter/b/a6a055/32.png) [@brijesh.kalavadia](https://community.graylog.org/u/brijesh.kalavadia)\
**Post date:** [October 26, 2022, 5:44pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/4 "2022-10-26T17:44:41Z")

</div>

> [@gsmith](#):
>
> set\_fields

Hi @gsmith ,  
Thanks for suggestion. I tried with set\_fields and it worked on above example… i just added prefix in "set\_fields(to\_map(nginx\_props),“nginx\_”)

but its fixing prefix which i know… however i want to add it dynamically… is there a way i can put some different logic to pick object and then set its prefix based on that value ?

sorry i am asking basic things as i am new to graylog and still learning things… if you can provide some example of that would be great

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [October 26, 2022, 9:29pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/5 "2022-10-26T21:29:19Z")

</div>

Hey @brijesh.kalavadia

> [@brijesh.kalavadia](#):
>
> however i want to add it dynamically… is there a way i can put some different logic to pick object and then set its prefix based on that value ?

For dynamically, I assume you referring to not having a pipeline and just ingesting the logs and “ **watch the magic work** ” 🙂 , If so perhaps a _new index_ and create a new _index template_ for those types of logs.

---

<div class="post-metadata">

**Author:** ![brijesh.kalavadia](https://avatars.discourse-cdn.com/v4/letter/b/a6a055/32.png) [@brijesh.kalavadia](https://community.graylog.org/u/brijesh.kalavadia)\
**Post date:** [October 26, 2022, 9:34pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/7 "2022-10-26T21:34:41Z")

</div>

Using pipeline only… I want to some logic to place where i have used “nginx” static value instead it picks dynamic value from json nested message.

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [October 26, 2022, 10:02pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/8 "2022-10-26T22:02:40Z")

</div>

hey,

> [@brijesh.kalavadia](#):
>
> is there a way i can put some different logic to pick object and then set its prefix based on that value ?

Only thing I can think of is using regex and/or a lookup table. Within the pipeline you can use the lookup table/s.

---

<div class="post-metadata">

**Author:** ![jivepig](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jivepig/32/13737_2.png) [@jivepig](https://community.graylog.org/u/jivepig)\
**Post date:** [October 31, 2022, 3:34pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/9 "2022-10-31T15:34:31Z")

</div>

Have you experimented with flatten\_json?  
Here’s a source and example with something I did to bring in logs into my test system.

```auto
rule "Random User Data Flatten Json Rule"
// From sample data : https://randomuser.me/api/
// Api input path: *
when
    true
then
    let sJson = to_string($message.result);
    let sJson = regex_replace(
        pattern: "^\\[|\\]$",
        value: sJson,
        replacement: ""
        );
    let rsJson = flatten_json(to_string(sJson), "flatten");
    set_fields(to_map(rsJson));
    remove_field("result");
    set_field("message", "parsed user data");
end
```

---

<div class="post-metadata">

**Author:** ![brijesh.kalavadia](https://avatars.discourse-cdn.com/v4/letter/b/a6a055/32.png) [@brijesh.kalavadia](https://community.graylog.org/u/brijesh.kalavadia)\
**Post date:** [October 31, 2022, 4:54pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/10 "2022-10-31T16:54:00Z")

</div>

Thanks @jivepig … I tried with flatten\_json and it works…

---

<div class="post-metadata">

**Author:** ![jivepig](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jivepig/32/13737_2.png) [@jivepig](https://community.graylog.org/u/jivepig)\
**Post date:** [November 1, 2022, 2:01pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/11 "2022-11-01T14:01:42Z")

</div>

Awesome! Do you have a sample rule that you used and what the output looked like? I’d love to see it.

---

<div class="post-metadata">

**Author:** ![brijesh.kalavadia](https://avatars.discourse-cdn.com/v4/letter/b/a6a055/32.png) [@brijesh.kalavadia](https://community.graylog.org/u/brijesh.kalavadia)\
**Post date:** [November 1, 2022, 9:29pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/12 "2022-11-01T21:29:25Z")

</div>

Single pipeline rule was good enough for us to get nested json field.

```auto
rule "extract json"
when 
    regex("(\\{.*\\})", to_string($message.message)).matches == true
then
   let json = regex("(\\{.*\\})", to_string($message.message), ["json"])["json"];
  // set_field("json", json);

set_fields(to_map(flatten_json(value: to_string(json), array_handler: "json")));

```

---

<div class="post-metadata">

**Author:** ![brijesh.kalavadia](https://avatars.discourse-cdn.com/v4/letter/b/a6a055/32.png) [@brijesh.kalavadia](https://community.graylog.org/u/brijesh.kalavadia)\
**Post date:** [November 3, 2022, 4:11pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/13 "2022-11-03T16:11:21Z")

</div>

it seems every field I am getting is type of string now… not sure why… it should be what field type itself… any suggestion ?

There are some fields we have which are not string and now it got converted to string.

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/b/b2f352fadf7d3ca8d08c484edc19639ac22a6f05.png)

---

<div class="post-metadata">

**Author:** ![jivepig](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jivepig/32/13737_2.png) [@jivepig](https://community.graylog.org/u/jivepig)\
**Post date:** [November 3, 2022, 5:48pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/14 "2022-11-03T17:48:51Z")

</div>

I think you will have to then set fields with their type after with something similar. When using flatten\_json, it will not set the field types at this time. You will need to:  
set\_field(“fieldname”, to\_type($messaage.fieldname));

---

<div class="post-metadata">

**Author:** ![brijesh.kalavadia](https://avatars.discourse-cdn.com/v4/letter/b/a6a055/32.png) [@brijesh.kalavadia](https://community.graylog.org/u/brijesh.kalavadia)\
**Post date:** [November 3, 2022, 5:59pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/15 "2022-11-03T17:59:08Z")

</div>

I am bit confuse here… will yo be able to help to correct in my below rule what you are suggesting.

```auto
rule "extract json"
when 
    regex("(\\{.*\\})", to_string($message.message)).matches == true
then
   let json = regex("(\\{.*\\})", to_string($message.message), ["json"])["json"];
  // set_field("json", json);

set_fields(to_map(flatten_json(value: to_string(json), array_handler: "json")));

```

---

<div class="post-metadata">

**Author:** ![brijesh.kalavadia](https://avatars.discourse-cdn.com/v4/letter/b/a6a055/32.png) [@brijesh.kalavadia](https://community.graylog.org/u/brijesh.kalavadia)\
**Post date:** [November 3, 2022, 6:05pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/16 "2022-11-03T18:05:45Z")

</div>

> [@jivepig](#):
>
> set\_field(“fieldname”, to\_type($messaage.fieldname));

There are no to\_type function

---

<div class="post-metadata">

**Author:** ![jivepig](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jivepig/32/13737_2.png) [@jivepig](https://community.graylog.org/u/jivepig)\
**Post date:** [November 3, 2022, 8:34pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/17 "2022-11-03T20:34:51Z")

</div>

can you paste in your raw log results? Replace names or IP’s or content with what you want. But let me see it for setting the fields?

---

<div class="post-metadata">

**Author:** ![brijesh.kalavadia](https://avatars.discourse-cdn.com/v4/letter/b/a6a055/32.png) [@brijesh.kalavadia](https://community.graylog.org/u/brijesh.kalavadia)\
**Post date:** [November 3, 2022, 8:44pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/18 "2022-11-03T20:44:02Z")

</div>

Below is raw log:

```auto
router-84d84bccc-rl8gk nginx: {"@timestamp": "2022-11-03T20:39:07+00:00", "source": "router", "nginx": {"remote_addr": "xx.xx.12.123", "remote_user": "39942", "body_bytes_sent": 0, "request_length": 656, "request_time": 0.464, "status": 202, "request": "PATCH /xxxxxx/emapi/v1/enablement/53815 HTTP/1.1", "request_method": "PATCH", "http_origin": "-", "http_referrer": "-", "site": "xxxxx.com", "port": 443, "http_user_agent": "python-requests/2.28.1" }}

```

and as a result after pipeline runs: We can see parsing is fine and it parse nested json however, every fields converted to type of string. however its not for all… for example… “nginx\_port” “nginx\_request\_length” these are not string fields…  
Also it looks like… flatten\_json only changing nested json field data type.

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/3/3550ba0ba36f227db8ba1b8e770714393171e326.png)

---

<div class="post-metadata">

**Author:** ![jivepig](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jivepig/32/13737_2.png) [@jivepig](https://community.graylog.org/u/jivepig)\
**Post date:** [November 4, 2022, 1:21pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/19 "2022-11-04T13:21:14Z")

</div>

I am opening an issue to investigate this a little bit further. How nested fields get parsed with flatten\_json is, there could be many different fields and types under that nested blob. The flatten\_json will parse all as strings. Performing set\_fields would need to be done on the fields requiring changes to non string type. I’ll open the issue just to make sure this is the case.To investigate:

---

<div class="post-metadata">

**Author:** ![brijesh.kalavadia](https://avatars.discourse-cdn.com/v4/letter/b/a6a055/32.png) [@brijesh.kalavadia](https://community.graylog.org/u/brijesh.kalavadia)\
**Post date:** [November 4, 2022, 3:54pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/20 "2022-11-04T15:54:32Z")

</div>

Thanks @jivepig for taking this further… just wondering is there a way i can track that issue and will there be any timeline which i can convey to our internal team ?  
Also any other workaround I can apply to have nested json works fine through pipeline ?

---

<div class="post-metadata">

**Author:** ![jivepig](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jivepig/32/13737_2.png) [@jivepig](https://community.graylog.org/u/jivepig)\
**Post date:** [November 4, 2022, 5:07pm UTC](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292/21 "2022-11-04T17:07:22Z")

</div>

> <https://github.com/Graylog2/graylog2-server/issues/13888>
>
> As per discussion in this :
> https://community.graylog.org/t/parsing-nested-json…-message-in-field-with-parent-object-in-pipeline/26292/19
> 
> \## Expected Behavior
> When using flatten\_json and uncovering different field types, flatten\_json to\_map should carry field types into the result.
> 
> \## Current Behavior
> Currently it appears flatten\_json converts all nested flattened json to string only when it has to flatten.
> 
> \## Possible Solution
> Carry field types through when creating parse to\_map
> 
> \## Steps to Reproduce (for bugs)
> Parse Nginx logs to json,
> rule "extract json"
> when 
> regex("(\\\\{.\*\\\\})", to\_string($message.message)).matches == true
> then
> let json = regex("(\\\\{.\*\\\\})", to\_string($message.message), \["json"\])\["json"\];
> // set\_field("json", json);
> 
> set\_fields(to\_map(flatten\_json(value: to\_string(json), array\_handler: "json")));
> 
> 
> 
> \## Context
> Can to\_map fields be carried forward when parsing with flatten\_json?
> 
> \## Your Environment
> 
> \* Graylog Version: 4.3
> \* Java Version: 11
> \* Elasticsearch Version: 7.10.2
> \* MongoDB Version: 4.0
> \* Operating System:n/a
> \* Browser version: n/a

[Next page](https://community.graylog.org/t/parsing-nested-json-message-in-field-with-parent-object-in-pipeline/26292.md?page=2)