Error - Replaced invalid timestamp value in message

[Burns]

1. Describe your incident:

I am getting the following errors:

message
Failed to process message with id ‘01GY0FPD4H69B4MH5FSBQFK2VN’: Replaced invalid timestamp value in message <68367902-daf0-11ed-9ac6-00505688713b> with current time

gl2_processing_error
Replaced invalid timestamp value in message <5c03e902-dafb-11ed-9ac6-00505688713b> with current time - Value <2023-04-14T15:34:22.200-0400> caused exception: Invalid format: “2023-04-14T15:34:22.200-0400” is malformed at “T15:34:22.200-0400”.

failure_details
Value <2023-04-14T14:15:57.854-0400> caused exception: Invalid format: “2023-04-14T14:15:57.854-0400” is malformed at “T14:15:57.854-0400”.

failure_cause
InvalidTimestampException

2. Describe your environment:
log ingest setup: (Graylog and Wazug-Indexer are on the same host)

Wazuh Agent > Graylog Input > Extractor > Stream

• OS Information:

Ubuntu 22.04

• Package Version:

Graylog Server 5.0.5-1
Graylog Enterprise 5.0.6-1
Wazuh Indexer 4.4.0-1

• Service logs, configurations, and environment variables:

All devices and servers are configure for Eastern time.

I was not able to find any configuration files that specified timestamp standards.

3. What steps have you already taken to try and solve the problem?

I’ve google searched this issue and looked around the Graylog forums. I did find many posts with a similar issue and I blieve this issue is the missing : in the timezone (0400) of the timestamp. I have tried figuring out where this timestamp is being generated and why it is invalid but have had no luck.

I created a JSON extractor but once that has crerated the new fields Graylog throws the error and replaces the timestamp which stops me from creating a new extracotr to fix it.

4. How can the community help?

I’m looking for some assistance in figuring out why this timestamp is invalid and the best why to resolve it.

Any help would be greatly appreciated as I have spent more time than I’d like to admit on troubleshooting this issue.

Thank you.

[gsmith]

Hey @Burns

If you need is to adjust the timestamp there are other means then to create an extractor.
Under every User ther is setting to adjust timezone. In Graylog configuration file you can adjust the timezone there.

As for log/s arriving with the incorrect timestamp a pipeline is preferred to adjust those but again there are other means to correct a issue like that.

@Burns

I created a JSON extractor but once that has crerated the new fields Graylog throws the error and replaces the timestamp which stops me from creating a new extracotr to fix it.

can you show this JSON extractor configuration?

[Burns]

Hello @gsmith,

Thank you for the reply.

I changed my user profile timezone and the error continued.

Here is a portion of the message coming in before the JSON extractor:

{“true”:1681830019.531832,“timestamp”:“2023-04-18T11:00:19.316-0400”, …

Here is the JSON extractor configuration:

Here is the extractor preview for the timestamp field:

I attempted to create a pipeline rule to adjust the timestamp:

rule “Timestamp Error”
when
contains(to_string($message.message), “timestamp”)
then
let new_date = parse_date(
value: to_string($message.TIMESTAMP),
pattern: “yyyy-MM-dd HH:mm:ss.SSS-”,
timezone: “America/Toronto”);
set_field(“timestamp”, new_date);
end

The rule is not working correctly and giving me the following error:

I’m not too expereinced with writting these and I’m not 100% what the original timestamp error is… I assumed it was the missing colon, but unsure.

Again, thank you for you assistance.

[gsmith]

Hey @Burns

Before modifying the timestamp, have you tried a different input? Just curious.

Here is an example. You may need to modify it fro your setup.

rule "XXX"
when
    has_field("message")
then
    let current_year = now().year;
    let ts_string = concat(to_string(current_year), concat(" ", to_string($message.timestamp)));
	let new_date = parse_date(to_string($message.timestamp),"yyyy MMM dd HH:mm:ss","America/Chicago");
	set_field("timestamp", new_date);
end

Just a note, Do you need to use JSON extractor? Have you tried a regex instead? The reason im asking is that REGEX extractor does have a converter, meaning you can be able to change the date/time.

Example.