Extract data from "message" into filebeat

Graylog Central (peer support)
1

Good morning everyone.
I’ve searched the forum a lot and also tried a lot, but I haven’t found any solution (I’m a newbie).
I have an ubuntu server (22.04) which sends filebeats to another ubuntu server with graylog 5.1 and Opensearch 2.8.

I would like to extract the date from the “message” field of the filebeat stream.

Blockquote
{
“filebeat_log_offset”: 1305062,
“filebeat_agent_name”: “”,
“filebeat_host_containerized”: false,
“gl2_remote_ip”: “”,
“gl2_remote_port”: ,
“source”: “”,
“beats_type”: “filebeat”,
“filebeat_host_os_platform”: “ubuntu”,
“gl2_source_input”: “64b77f3dd9427001a863d4c5”,
“filebeat_@metadata_beat”: “filebeat”,
“filebeat_host_os_version”: “22.04.2 LTS (Jammy Jellyfish)”,
“filebeat_host_os_kernel”: “5.15.0-76-generic”,
“filebeat_host_os_codename”: “jammy”,
“filebeat_@timestamp”: “2023-07-27T13:16:32.941Z”,
“filebeat_agent_type”: “filebeat”,
“filebeat_@metadata_version”: “8.8.2”,
“filebeat_host_name”: “”,
“gl2_source_node”: “354bd815-206b-41e2-b990-36a5964f6dfd”,
“filebeat_agent_version”: “8.8.2”,
“timestamp”: “2023-07-27T13:16:32.941Z”,
“filebeat_agent_ephemeral_id”: “147295d8-ca1a-490e-bd51-fe91c03d4d40”,
“gl2_accounted_message_size”: 1282,
“filebeat_host_mac”: [
“”
],
“filebeat_input_type”: “filestream”,
“filebeat_host_os_name”: “Ubuntu”,
“streams”: [
“000000000000000000000001”
],
“gl2_message_id”: “01H6BQWVSD100K1V1FWEFPQMPE”,
“filebeat_host_os_family”: “debian”,
“message”: "Jul 26 10:34:09 kernel: [96548.858344] [TEST]: IN=ens1 OUT= MAC=…",
“filebeat_ecs_version”: “8.0.0”,
“filebeat_host_ip”: [
“”
],
“filebeat_host_architecture”: “x86_64”,
“filebeat_@metadata_type”: “_doc”,
“filebeat_host_os_type”: “linux”,
“filebeat_host_hostname”: “”,
“filebeat_host_id”: “7f52aa3a4d2844d29e750f51227265dd”,
“filebeat_agent_id”: “749611d7-5d5a-4d14-9628-7c0d7f9f1764”,
“_id”: “d4981733-2c7f-11ee-b6f0-0002207b288f”,
“filebeat_log_file_path”: “/var/log/syslog”
}
Blockquote

In bold the timestamp of filebeat and that of the real message
I’m using a pipelined rule, but what I get is the received date and time and not that of the message.

the rule

Blockquote
rule “Test”
when
has_field(“message”)
then
let mes = to_string($message.message);
let cp = grok (pattern:“%{TIMESTAMP_ISO8601:datetime}”, value: mes);
set_field(“tm_key”, cp);
end
Blockquote

But the result is the timestamp of filebeat and not that of the message

Blockquote
tm_key
{“datetime”:“2023-07-27T13:16:32.941Z”,“YEAR”:“2023”,“MONTHNUM”:“07”,“MONTHDAY”:“27”,“HOUR”:“13”,“MINUTE”:“16”,“SECOND”:“32.941”,“ISO8601_TIMEZONE”:“Z”}
Blockquote

I hope someone can put me on the right track.

Thanks a lot everyone.

2

this is not a TIMESTAMP_ISO8601, so your GROK will not work.
Next possible steps for you:
a) change log file settings to write TIMESTAMP_ISO8601 timestamps
or
b) try to GROK your timestamp format, magically extend it with year and timezone, then do something like I do in some of my pipelines to get the real timestamp into SEARCH:
set_field(“timestamp”, parse_date(value: to_string(groked_message.logdate), pattern: “yyyy-MM-dd’T’HH:mm:ss.SSSZ”));

3

Thanks a lot. :grinning:
I try immediately.
Though I was testing an extractor that seems to work.

4

Ok, unfortunately, with the pipeline rule I couldn’t make it, but I solved it with an extractor.
I only have the problem of determining the weekend, the extractor gives me the date and time but not the day of the week.
I searched the forum but the solutions work only with the date of receipt of the log and not with that of the message contained.

5

Hello everyone.
After a day of testing I got the following result.
With a rule in the pipeline

when
    has_field(message)
then
    let gg = to_long(to_date($message.timestamp)dayOfWeek);
    set_field("w_day", gg);
end

I created the w_day field which contains the day of the week number.
Now with a new rule, or the same, in the pipeline I would like to turn the number into text
for example:

when
      w_day == 2
then
      set_field("Day","Wednesday")
end

I’ve tried everything but I can’t understand the syntax of regex or grok (I’m not a programmer).

I need a help!

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.