Parsing nested json message in field with parent object in pipeline

Graylog Central (peer support)
22

I wanted to add to this. As an example, after you have processed to map, you could set the fields by type. As an example,
set_field(“nginx_body_bytes_sent”,to_double($message.nginx_body_bytes_sent))

More examples here on how to set the field types, look to conversion
https://docs.graylog.org/docs/functions-by-category-1

23

This will not help… we want field to be set dynamically … because I shared example of nginx but we have others too… so setting up field type for each and every field by this suggestion will not help… correct me if i am wrong.

1 Like
24

Ok so meanwhile i tried below and it seems it only converted “nginx_body_bytes_sent” field type and kept rest of the field as it is “string”… am I doing anything wrong ?

rule "extract json"
when 
    regex("(\\{.*\\})", to_string($message.message)).matches == true
then
   let json = regex("(\\{.*\\})", to_string($message.message), ["json"])["json"];
  // set_field("json", json);

set_fields(to_map(flatten_json(value: to_string(json), array_handler: "json")));
set_field("nginx_body_bytes_sent",to_double($message.nginx_body_bytes_sent));
set_field("nginx_bytes_sent",to_double($message.nginx_bytes_sent));
set_field("nginx_port",to_long($message.nginx_port));
set_field("nginx_request_length",to_double($message.nginx_request_length));
set_field("nginx_request_time",to_double($message.nginx_request_time));
set_field("nginx_site",to_ip($message.nginx_site));
set_field("nginx_status",to_long($message.nginx_status));

end
25

Updates to the issue in Github. Let’s see what dev says with a quick look.

26

you want me to add this as well in issue in github as well ?

27

That would be great. Most of it is already there but please do.

28

Remember that you cannot change the type of an existing field in Elastic.

29

yes but flatten_json() itself changing original type to string… we want original type when flatten_json() parse fields.

30

Understood. I am working on it for the next bugfix release.

Holen Sie sich Outlook für Android

1 Like
31

Hello Team,
I go notification that flatten_json converts all values in that nested blob to string · Issue #13888 · Graylog2/graylog2-server · GitHub is closed so may I know if fix will be available in current version or any next version ? and how would I fix from my side so that we can continue with pipeline…

32

As mentioned it will be available in the next 4.3 Bugfix release. Also in 5.0. You could install the 5.0 RC if you don’t want to wait.

Holen Sie sich Outlook für Android

33

sorry to ask but where can I find those bugfix and RC release ? I probably try on our Dev setup before we do upgrade on our prod setup .

34

5.0 RC is not yet released, but coming soon.

35

Is there any timeline for those releases ?

36

Bugfix releases are generally produced in the first week of every month, barring unusual circumstances.

37

Hi @brijesh.kalavadia ,
I have tested this new rule with the data in GL V5 RC2 and This change creates the option to parse as strings true or false.

rule "Random User Data Flatten Json Rule"
// From sample data : https://randomuser.me/api/
// Api input path: *
when
    true
then
    let sJson = to_string($message.result);
    let sJson = regex_replace(
        pattern: "^\\[|\\]$",
        value: sJson,
        replacement: ""
        );
//  let rsJson = flatten_json(to_string(sJson), "flatten");
    let rsJson = flatten_json(value: sJson, array_handler: "flatten", stringify: false);  
    set_fields(to_map(rsJson));
    remove_field("result");
    set_field("message", "parsed user data");
end
1 Like
38

Thanks @jivepig for sharing updates… really appreciate for quick fix… I am waiting for 4.3 bug fix release … once its out … i will test as you suggested.

1 Like
39

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.