Unable to use pipeline for nested json

Graylog Central (peer support)
1

Hey,

I want to pipeline to set fields for the below mentioned json .However I am unable to do so.

Json example:
{“Message”:“124”, “data”:{“MT”:“msgtype”, “UID”:“123abc”}}

I need set fields for the inner json fields MT and UID along with message.

Also I have some json structure where values are not string in json.

Json example:
{“Message”:“124”, “data”:{“MT”:“msgtype”, “Qty”:23}}
{“Message”:“124”, “Data”:123}
Unable to set fields for this message.

I have tired to select_jsonpath and parse_json.

Kindly let me know how can I write pipeline for these example.

2

Try to use this pipeline rule, it should extract also inner json:

rule "extract-json"
when
    starts_with(to_string($message.message), "{") && ends_with(to_string($message.message), "}")
then
    let json = parse_json(to_string($message.message));
    let map = to_map(json);
    set_fields(map);
end
3

I had tired to use the above example but did not get the correct output.
Can I get the parameter Message, MT and UID as set fields with the above rule you have shared.
Json example:
{“Message”:“124”, “data”:{“MT”:“msgtype”, “UID”:“123abc”}}

4

If I undestand corretly, do you want field names MT, UID? Because my example extract its as data_MT, data_UID.

Check screenshot, how my snippet parsed your data.
json-extract

5

Thanks I will try data_MT should also be fine

6

I have attached the image for the pipeline and data.
I am getting the same data and fields are not set :
Kindly let me know if I have entered anything incorrectly.

image
image831×227 65.8 KB

image

7

You used non standard quotes “ not ", so pipeline rule doesn’t match at all. Change to normal double quotes, don’t copy and paste from this websites, as it’s changing it to non standard.

8

Thanks you for the update.

Even if I dnt copy and paste it. I dnt get the desired output.
I am unable to data_MT and data_UID

20200424_210635

Kindly help me with the solution.

9

I am still unable to get the inner json data.

10

I tried it on version graylog 3.2.4, with only one pipeline. Check if you don’t have pipeline, or extactors, that override json parsing.

11

I am using the same graylog version.
Using this pipeline I get message and data fields but unable to get the fields data_UID and data_MT as seen in your image.

Can you share your pipeline details and if any modifications done by you in json parsing.

12

Sorry for error, but i found that actual json inner extraction was done using json extractor, and not pipeline rule at all. I used this extractor on Input for json:

{
  "extractors": [
    {
      "title": "json",
      "extractor_type": "json",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "",
      "extractor_config": {
        "flatten": false,
        "list_separator": ", ",
        "kv_separator": ":",
        "key_prefix": "",
        "key_separator": "_",
        "replace_key_whitespace": false,
        "key_whitespace_replacement": "_"
      },
      "condition_type": "none",
      "condition_value": ""
    }
  ],
  "version": "3.2.4"
}
13

Thank you for sharing the details.
Can this be done using pipeline rules?

14

Maybe this could work for you, if you want parse only data {} content:

rule "extract-json2"
when
    starts_with(to_string($message.message), "{") && ends_with(to_string($message.message), "}")
then
    let ex = regex(pattern: "\"data\":(\\{.*?\\})", value: to_string($message.message));
    let json = parse_json(to_string(ex["0"]));
    set_fields(to_map(json));
end
15

I want all the fields from the json including the inner json which I am unable to get.

16

Hi,

I have a log line the following format:

{
“Details”:“request”,
“Other”:{
“MT”:“123”,
“St”: 2,
“Ip”:“192.168.0.9”
}
}

I have applied select_jsonpath to set field from json.
I need to use lookup table to get some values like parameter mt value 123 i need to replace from lookup to user1 and IP value to host name.

I want to use select_jsonpath to pipeline json fields.kindly help with the solution to implement lookup along with select_jsonpath.

Can you please help. If you need any other information let me know

17

Basically my issue is I have unable to read the inner json data on pipeline because of which lookup is not working

18

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.