Graylog missing field in json embedded object

Graylog Central (peer support)
,
1

Hello,
I’m shipping logfiles in json format using logstash and its json filter.
Example (status is the important object)
{“status”: {“errorDetails”: “”, “errorMessage”: “”, “statusCode”: “SUCCESS”, “error”: false}, “logTag”: “SNOWFLAKE_QUERY”, “context”: {“datasetId”: “CSCOKE436PI_UAT”}, “executionMetrics”: {“execMillis”: 4888, “execType”: “Extract”, “execTarget”: “CS_OD_RPT”, “execTechnology”: “Snowflake”}, “logLevel”: “INFO”, “hostName”: “mue2rhepaggq001”, “dateTime”: “02/14/2020 07:26:32.695 UTC”, “logType”: “EVENT”, “version”: “362156”, “sourceApplication”: {“applicationName”: “Connect Extracts”, “buildVersion”: “CS_6.9.7.19”, “applicationComponent”: “FileGen”}, “messageId”: “4”, “duration”: {“totalTime”: 4888, “timeUnit”: “MILLISECONDS”}, “transactionId”: “4a691073-a63f-44cd-8067-eae2ea0a3b4d”, “dataVolumetrics”: “NA”, “event”: {“action”: “FileGen”, “type”: “Extract”, “target”: “Connect Extracts”}}

What I see in the logstash debug output:
“status” => {
“errorMessage” => “”,
“errorDetails” => “”,
“statusCode” => “SUCCESS”,
“error” => false
}

In Graylog I get status_statusCode only
What I expect is also status_error

Why don’t I get status_error ?

2

he @kai_poitschke

when you have logstash in between - how did you send that messages from logstash to graylog? What input did you use, how is logstash configured?

3

Hello,
the logstash version is 6.4.2
We ship via gelf

config looks like this ( removed stuff for clarity)

input {
  file {
    path => "/.../filegen_eventlog*.json"
    start_position => "beginning"
    type => "eventlog-extract-filegen"

    add_field => { "source_env" => "usprod" }

    close_older => 600
    max_open_files => 100
  }
}

filter {
    json {
      source => "message"
    }
    date {
      locale => "en"
      match => [ "dateTime" ,
                "ISO8601"
                ]
    }
  }
}
output {
  gelf {
    host => "....."
    port => "12201"
    protocol => "TCP"
  }
 # stdout { codec => "rubydebug" }
}
4

he @kai_poitschke

as you do the json processing in logstash and only send GELF as output you need to configure logstash that the nested json arrays are used. I do not know about the json parsing in logstash - but maybe someone else is able to give some light here.

5

Hello Jan,
logstash is sending it right, see my first posting. The status_statusCode field is present in Graylog (and elasticsearch), but status_error is not visible.

6

follow the trail - step by step. is that field is transfered from logstash to graylog? from graylog stored in elasticsearch?

Use the processing pipelines and the debug function to find the problem.

7

Hello @Jan,
please provide or point me to an example how to debug this in the pipeline. I’m not familiar in debugging GL.

8

Check documentation, I think it is very clear:
https://docs.graylog.org/en/3.2/pages/pipelines/functions.html#debug

  1. So use this snippet in your pipeline rule, or modify:

let debug_message = concat("Dropped message from ", to_string($message.source));
debug(debug_message);

  1. And after that check graylog log file to see debug output;
    sudo tail -f /var/log/graylog-server/server.log
9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.