Hello Team,
I have the following Problem and would love to have some help 1. Describe your incident:
I’m receiving docker logs in my graylog-instance and I want to extract the date from the message field and add it to a new field called timestamp_message_created field.
The part of my message, that is interesting looks like the following:
2023-05-09 16:24:01.9051|
And I’m using the following pipeline rule for the processing:
rule “Rule_1”
when
has_field(“message”)
then
let pattern = "%{DATA:timestamp}\|;
let matches = grok(pattern, to_string($message.message));
let date_string = to_string(matches[“timestamp”]);
let created_date = parse_date(date_string,“yyyy-MM-dd HH:mm:ss.SSSS”, “CEST”, “Europe/Zurich”);
set_field(“timestamp_message_created”, created_date);
end
Basically this works fine, but the problem is, that my field that I’m adding has afterwards the data type string and not date and my parsed date looks like this:
2023-05-09T16:24:01.905Z
Is there a way, how I can succefully parse this string in a valid date format and add it to my new field?
2. Describe your environment:
I’m using graylog version 4.3, alongside the elasticsearch version 7.10.2 and the mongo version 4.2
My logs are generated by NLog
3. What steps have you already taken to try and solve the problem?
I have already tried different timezone, patterns and also the flex_parse_date() method, but neither of them was working as expected.
A quick preamble regarding elaticsearch (or OpenSearch) field mappings: the default behavior for field types is Dynamic Field Mappings. This dynamic field mapping only occurs once, when the first message containing that field is saved to an index. After this occurs, the field mapping cannot be changed. This means that if a field is dynamically “typed” as string, it will always be string in that index, no matter what other data is detected in the future. The only way to “try again” with dynamic mappings, for example if you change the formatting of the text for a given field, is to rotate the index in graylog, effectively creating a new index.
With that out of the way, it appears that based on elastic’s documentation about date detection, a field will be dynamically “typed” as date if the date string matches the “built in formats” for dates.
Regarding the formatting of the date as a string value, graylog pipeline function parse_date only returns a DateTime type which does not appear to be your intended output format.
You can, however, use the format_date function , for example:
