Pipeline rule on ingested JSON data via Filebeat and NDJSON parser

[foobug]

Hello everyone,

for a project I put certain logs (access logs, applications logs (log4j), audit logs) of Atlassian Jira into Graylog.
At the end nothing special simply parsing the logs with GROK pattern, replacing timestamp with origin timestamp, replace square brackets.

I have an Elasticsearch & Kibana background where I did all of this as well in a previous project.

But there is one last thing that I can’t solve / replace.

I’ve ingesting data from logs that contain JSON data with the following Filebeat input:

- type: filestream
  id: atlassian-jira-audit
  encoding: utf-8
  paths:
    - "/var/atlassian/application-data/jira/log/audit/*.audit.log"
  ignore_older: 24h
  fields_under_root: true
  fields:
    log.file.name: atlassian-jira-audit.log
    type: audit
  parsers:
    - ndjson:
        target: "application.audit"
        add_error_key: true
        expand_keys: true

Example entry from the source JSON file:

...
{"affectedObjects":[],"auditType":{"action":"Support Zip Created","actionI18nKey":"stp.audit.summary.support-zip.created","area":"ECOSYSTEM","category":"System","categoryI18nKey":"stp.audit.category.system","level":"BASE"},"author":{"id":"-1","name":"System","type":"system"},"changedValues":[],"extraAttributes":[],"method":"Unknown","system":"https://jira.example.com","timestamp":{"epochSecond":1678438469,"nano":750000000},"version":"1.0"}
...

The data is stored in Graylog as expected (viewed from search). Just the . are replaced with _
(Fields with dots are inaccessible in the pipeline processor · Issue #6588 · Graylog2/graylog2-server · GitHub)

I’ve created a pipeline rule to parse the epochSecond value and set it as timestamp.

rule "Jira | Replace @timestamp with application.audit.timestamp.epochSecond for audit logs"
when
    true
then
    debug(to_string($message.`application.audit.timestamp.epochSecond`));
    debug(to_string($message.application_audit_timestamp_epochSecond));
    let origin_event_date = flex_parse_date(to_string($message.`application.audit.timestamp.epochSecond`));
    debug("origin_event_date: " + to_string(origin_event_date));
    set_field("timestamp", origin_event_date);
end

But I can’t figure out how the access the field, my result is always:

2023-03-13T19:07:54.511+01:00 INFO  [Function] PIPELINE DEBUG: Passed value is NULL.
2023-03-13T19:07:54.514+01:00 INFO  [Function] PIPELINE DEBUG: Passed value is NULL.
2023-03-13T19:07:54.514+01:00 INFO  [Function] PIPELINE DEBUG: origin_event_date:

Environment:

• OS Information: Ubuntu 20 LTS
• Package Version: Graylog 4.2.13
• Message Processors Configuration: Message Filter Chain BEFORE Pipeline Processor

Any ideas welcome.

Kind Regards,
Foobug

[gsmith]

Hello && Welcome @foobug

Correct me if I’m wrong you need a pipe for epoch time? if so here is an example

rule "Jira | Replace @timestamp with application.audit.timestamp.epochSecond for audit logs"
when
    true
then
  let ts_millis = to_long($message.application.audit.timestamp.epochSecond) / 1000000;
  let new_date = parse_unix_milliseconds(ts_millis);
  set_field("epoch_timestamp", new_date);
  set_field("timestamp", new_date);
end

divide unix timestamp in nanoseconds by 1000000 and you have miliseconds. Then use it in function parse_unix_milliseconds()

Some findings;

• Graylog 4, apache, epoch microseconds to date - #2 by shoothub
• Epoch time to readable datetime stamp - #8 by jochen
• Issue with epoch timestamp being converted to datetimestamp · Issue #2409 · Graylog2/graylog2-server · GitHub

[foobug]

Hello @gsmith,

at the end yes, I need a pipe for epoch time but the problem is still that the pipe isn’t reading the values that are parsed and visible in the search.

I’ve also checked your example and it has the same result:

2023-03-14T08:33:10.756+01:00 INFO  [Function] PIPELINE DEBUG: ts_millis: 0