Parsing JSON format with Filebeat

Graylog Central (peer support)
1

Hi
I am having a little trouble understanding how the parsing of JSON format works, when using filebeat as a collector. I have gone through a few forum posts and docs and can’t seem to get things looking right.

Currently the format of the string more or less looks like this:

{"timestamp":"2024-11-13T07:32:51.065840Z","level":"DEBUG","fields":{"message":"🔚 Dropping engine manager"},"filename":"modules/machine_vision/crates/machine_learning/src/engines/yolo.rs"}

There are also a logs where there are a few more keys within fields, but this is the most basic format.

I have done a lot of playing around with the filebeat config, and it currently looks like this:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}


output.logstash:
   hosts: ["${user.graylog_host}:5044"]
path:
   data: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/data
   logs: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/log

filebeat.inputs:

- type: filestream
  id: dynamic-file-tracker
  paths:
    - /home/**/logs/**/
  
  parsers:
    - ndjson:
        target: ""
        add_error_key: true
        overwrite_keys: true
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: fields.message
  json.overwrite_keys: true
  

processors:
  - decode_json_fields:
      fields: ["timestamp", "fields", "filename", "level"]
      process_array: false
      max_depth: 2
      target: ""
      overwrite_keys: true
      add_error_key: true
  

close_inactive: 5m              
scan_frequency: 10s

The files are saved as system files and not with the .json extension.

And the outputted fields I get looks like this:

image
image1641×985 60.1 KB

The things I am trying to fix:

  1. The timestamp is coming from the time in which the log is being read, and not coming from the log itself, and I want to be able to replace it
  2. I am trying to replace the ‘message’ field with the ‘field_message’ field.
  3. When there are more fields within the “fields” key, then I want them to be disectted as there key only, not having field as a prefix. Eg: plugin field lives within “fields” and is displayed as “fields_plugin” not just plugin (This one I have to fix with pipelines, so the first two are the most important)
  4. I am trying to have this constantly import new logs as new lines are added to the file, to avoid repeated information

Any help in the right direction would be amazing

2

Hey @mromeo

Something like the below as a rule should help with the first two problems.

    set_field("message", to_string($message.field_message));
    let new_date = parse_date(to_string($message.@timestamp), "yyyy-MM-dd HH:mm:ss,SSS");
    set_field("timestamp", new_date);