# Unable to use pipeline for nested json

**URL:** <https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118>\
**Category:** Graylog Central (peer support)\
**Tags:** pipeline-rules\
**Created:** [April 23, 2020, 8:11pm UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118 "2020-04-23T20:11:04Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![gitdish](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gitdish/32/6302_2.png) [@gitdish](https://community.graylog.org/u/gitdish)\
**Post date:** [April 23, 2020, 8:11pm UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/1 "2020-04-23T20:11:05Z")

</div>

Hey,

I want to pipeline to set fields for the below mentioned json .However I am unable to do so.

Json example:  
{“Message”:“124”, “data”:{“MT”:“msgtype”, “UID”:“123abc”}}

I need set fields for the inner json fields MT and UID along with message.

Also I have some json structure where values are not string in json.

Json example:  
{“Message”:“124”, “data”:{“MT”:“msgtype”, “Qty”:23}}  
{“Message”:“124”, “Data”:123}  
Unable to set fields for this message.

I have tired to select\_jsonpath and parse\_json.

Kindly let me know how can I write pipeline for these example.

---

<div class="post-metadata">

**Author:** ![shoothub](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/shoothub/32/6412_2.png) [@shoothub](https://community.graylog.org/u/shoothub)\
**Post date:** [April 24, 2020, 8:17am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/2 "2020-04-24T08:17:40Z")

</div>

Try to use this pipeline rule, it should extract also inner json:

```auto
rule "extract-json"
when
    starts_with(to_string($message.message), "{") && ends_with(to_string($message.message), "}")
then
    let json = parse_json(to_string($message.message));
    let map = to_map(json);
    set_fields(map);
end

```

---

<div class="post-metadata">

**Author:** ![gitdish](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gitdish/32/6302_2.png) [@gitdish](https://community.graylog.org/u/gitdish)\
**Post date:** [April 24, 2020, 8:21am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/3 "2020-04-24T08:21:22Z")

</div>

I had tired to use the above example but did not get the correct output.  
Can I get the parameter Message, MT and UID as set fields with the above rule you have shared.  
Json example:  
{“Message”:“124”, “data”:{“MT”:“msgtype”, “UID”:“123abc”}}

---

<div class="post-metadata">

**Author:** ![shoothub](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/shoothub/32/6412_2.png) [@shoothub](https://community.graylog.org/u/shoothub)\
**Post date:** [April 24, 2020, 8:29am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/4 "2020-04-24T08:29:49Z")

</div>

If I undestand corretly, do you want field names MT, UID? Because my example extract its as data\_MT, data\_UID.

Check screenshot, how my snippet parsed your data.  
 ![json-extract](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/e/e4504ef34e8227b5e91c6234e9869448664ad295.png)

---

<div class="post-metadata">

**Author:** ![gitdish](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gitdish/32/6302_2.png) [@gitdish](https://community.graylog.org/u/gitdish)\
**Post date:** [April 24, 2020, 8:32am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/5 "2020-04-24T08:32:00Z")

</div>

Thanks I will try data\_MT should also be fine

---

<div class="post-metadata">

**Author:** ![gitdish](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gitdish/32/6302_2.png) [@gitdish](https://community.graylog.org/u/gitdish)\
**Post date:** [April 24, 2020, 9:10am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/6 "2020-04-24T09:10:23Z")

</div>

I have attached the image for the pipeline and data.  
I am getting the same data and fields are not set :  
Kindly let me know if I have entered anything incorrectly.

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/e/e6df41c5d5d85694190987d2f7c500cd8f0ca2e8.png)  
 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/b/bda46e46b3967bbb92827cea8f21fb18babfac99.png)

---

<div class="post-metadata">

**Author:** ![shoothub](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/shoothub/32/6412_2.png) [@shoothub](https://community.graylog.org/u/shoothub)\
**Post date:** [April 24, 2020, 11:15am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/7 "2020-04-24T11:15:10Z")

</div>

You used non standard quotes `“` not `"`, so pipeline rule doesn’t match at all. Change to normal double quotes, don’t copy and paste from this websites, as it’s changing it to non standard.

---

<div class="post-metadata">

**Author:** ![gitdish](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gitdish/32/6302_2.png) [@gitdish](https://community.graylog.org/u/gitdish)\
**Post date:** [April 24, 2020, 3:38pm UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/8 "2020-04-24T15:38:25Z")

</div>

Thanks you for the update.

Even if I dnt copy and paste it. I dnt get the desired output.  
I am unable to data\_MT and data\_UID

![20200424_210635](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/c/c41881aae6c9c3616292617540b1c0fd89d9d212.jpeg)

Kindly help me with the solution.

---

<div class="post-metadata">

**Author:** ![gitdish](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gitdish/32/6302_2.png) [@gitdish](https://community.graylog.org/u/gitdish)\
**Post date:** [April 27, 2020, 6:38am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/9 "2020-04-27T06:38:02Z")

</div>

I am still unable to get the inner json data.

---

<div class="post-metadata">

**Author:** ![shoothub](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/shoothub/32/6412_2.png) [@shoothub](https://community.graylog.org/u/shoothub)\
**Post date:** [April 27, 2020, 7:15am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/10 "2020-04-27T07:15:51Z")

</div>

I tried it on version graylog 3.2.4, with only one pipeline. Check if you don’t have pipeline, or extactors, that override json parsing.

---

<div class="post-metadata">

**Author:** ![gitdish](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gitdish/32/6302_2.png) [@gitdish](https://community.graylog.org/u/gitdish)\
**Post date:** [April 27, 2020, 8:28am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/11 "2020-04-27T08:28:23Z")

</div>

I am using the same graylog version.  
Using this pipeline I get message and data fields but unable to get the fields data\_UID and data\_MT as seen in your image.

Can you share your pipeline details and if any modifications done by you in json parsing.

---

<div class="post-metadata">

**Author:** ![shoothub](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/shoothub/32/6412_2.png) [@shoothub](https://community.graylog.org/u/shoothub)\
**Post date:** [April 27, 2020, 9:01am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/12 "2020-04-27T09:01:04Z")

</div>

Sorry for error, but i found that actual json inner extraction was done using json extractor, and not pipeline rule at all. I used this extractor on Input for json:

```auto
{
  "extractors": [
    {
      "title": "json",
      "extractor_type": "json",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "",
      "extractor_config": {
        "flatten": false,
        "list_separator": ", ",
        "kv_separator": ":",
        "key_prefix": "",
        "key_separator": "_",
        "replace_key_whitespace": false,
        "key_whitespace_replacement": "_"
      },
      "condition_type": "none",
      "condition_value": ""
    }
  ],
  "version": "3.2.4"
}

```

---

<div class="post-metadata">

**Author:** ![gitdish](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gitdish/32/6302_2.png) [@gitdish](https://community.graylog.org/u/gitdish)\
**Post date:** [April 27, 2020, 9:04am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/13 "2020-04-27T09:04:42Z")

</div>

Thank you for sharing the details.  
Can this be done using pipeline rules?

---

<div class="post-metadata">

**Author:** ![shoothub](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/shoothub/32/6412_2.png) [@shoothub](https://community.graylog.org/u/shoothub)\
**Post date:** [April 27, 2020, 9:45am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/14 "2020-04-27T09:45:47Z")

</div>

Maybe this could work for you, if you want parse only data {} content:

```auto
rule "extract-json2"
when
    starts_with(to_string($message.message), "{") && ends_with(to_string($message.message), "}")
then
    let ex = regex(pattern: "\"data\":(\\{.*?\\})", value: to_string($message.message));
    let json = parse_json(to_string(ex["0"]));
    set_fields(to_map(json));
end

```

---

<div class="post-metadata">

**Author:** ![gitdish](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gitdish/32/6302_2.png) [@gitdish](https://community.graylog.org/u/gitdish)\
**Post date:** [April 27, 2020, 9:58am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/15 "2020-04-27T09:58:15Z")

</div>

I want all the fields from the json including the inner json which I am unable to get.

---

<div class="post-metadata">

**Author:** ![gitdish](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gitdish/32/6302_2.png) [@gitdish](https://community.graylog.org/u/gitdish)\
**Post date:** [May 8, 2020, 10:38am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/16 "2020-05-08T10:38:23Z")

</div>

Hi,

I have a log line the following format:

{  
“Details”:“request”,  
“Other”:{  
“MT”:“123”,  
“St”: 2,  
“Ip”:“192.168.0.9”  
}  
}

I have applied select\_jsonpath to set field from json.  
I need to use lookup table to get some values like parameter mt value 123 i need to replace from lookup to user1 and IP value to host name.

I want to use select\_jsonpath to pipeline json fields.kindly help with the solution to implement lookup along with select\_jsonpath.

Can you please help. If you need any other information let me know

---

<div class="post-metadata">

**Author:** ![gitdish](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gitdish/32/6302_2.png) [@gitdish](https://community.graylog.org/u/gitdish)\
**Post date:** [May 8, 2020, 11:36am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/17 "2020-05-08T11:36:08Z")

</div>

Basically my issue is I have unable to read the inner json data on pipeline because of which lookup is not working

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [May 22, 2020, 11:36am UTC](https://community.graylog.org/t/unable-to-use-pipeline-for-nested-json/15118/18 "2020-05-22T11:36:13Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.