Hello everyone,
for a project I put certain logs (access logs, applications logs (log4j), audit logs) of Atlassian Jira into Graylog.
At the end nothing special simply parsing the logs with GROK pattern, replacing timestamp with origin timestamp, replace square brackets.
I have an Elasticsearch & Kibana background where I did all of this as well in a previous project.
But there is one last thing that I can’t solve / replace.
I’ve ingesting data from logs that contain JSON data with the following Filebeat input:
- type: filestream
id: atlassian-jira-audit
encoding: utf-8
paths:
- "/var/atlassian/application-data/jira/log/audit/*.audit.log"
ignore_older: 24h
fields_under_root: true
fields:
log.file.name: atlassian-jira-audit.log
type: audit
parsers:
- ndjson:
target: "application.audit"
add_error_key: true
expand_keys: true
Example entry from the source JSON file:
...
{"affectedObjects":[],"auditType":{"action":"Support Zip Created","actionI18nKey":"stp.audit.summary.support-zip.created","area":"ECOSYSTEM","category":"System","categoryI18nKey":"stp.audit.category.system","level":"BASE"},"author":{"id":"-1","name":"System","type":"system"},"changedValues":[],"extraAttributes":[],"method":"Unknown","system":"https://jira.example.com","timestamp":{"epochSecond":1678438469,"nano":750000000},"version":"1.0"}
...
The data is stored in Graylog as expected (viewed from search). Just the .
are replaced with _
(Fields with dots are inaccessible in the pipeline processor · Issue #6588 · Graylog2/graylog2-server · GitHub)
I’ve created a pipeline rule to parse the epochSecond value and set it as timestamp.
rule "Jira | Replace @timestamp with application.audit.timestamp.epochSecond for audit logs"
when
true
then
debug(to_string($message.`application.audit.timestamp.epochSecond`));
debug(to_string($message.application_audit_timestamp_epochSecond));
let origin_event_date = flex_parse_date(to_string($message.`application.audit.timestamp.epochSecond`));
debug("origin_event_date: " + to_string(origin_event_date));
set_field("timestamp", origin_event_date);
end
But I can’t figure out how the access the field, my result is always:
2023-03-13T19:07:54.511+01:00 INFO [Function] PIPELINE DEBUG: Passed value is NULL.
2023-03-13T19:07:54.514+01:00 INFO [Function] PIPELINE DEBUG: Passed value is NULL.
2023-03-13T19:07:54.514+01:00 INFO [Function] PIPELINE DEBUG: origin_event_date:
Environment:
- OS Information: Ubuntu 20 LTS
- Package Version: Graylog 4.2.13
- Message Processors Configuration: Message Filter Chain BEFORE Pipeline Processor
Any ideas welcome.
Kind Regards,
Foobug