I’ve read through a number of threads/topics/solutions both in the community and general google but I still can’t get a working solution.
I have been unable to replace the graylog ingestion timestamp with the log message’s timestamp.
I have tried using
- an extractor using COMMONAPACHELOG with the HTTPDATE going to timestamp field. This had no impact the timestamp was equal to the Timestamp (log ingestion time).
- modified extractor by expanding the COMMONAPACHELOG grok pattern and writing HTTPDATE to log_timestamp which showed the correct value in log_timestamp as “28/Apr/2014:18:23:21 +1000”.
- I then added a pipeline with the pipeline rule
rule “substitute ingest time to log_time”
when
has_field(field: “log_timestamp”)
then
let new_time = parse_date(to_string($message.log_timestamp),“dd/MM/yyyy:HH:mm:ss”);
set_field(“timestamp”, new_time);
end
This did not change the timestamp field
- I then disconnected the pipeline and tried adding a second extractor to the COMMONAPACHELOG with the field going to log_timestamp. This second extractor set up with “source field” of log_timestamp the “store as field” set to timestamp, a converter of “Convert date type” with format string dd-MMM-yyyy:HH:mm:ss Z with no value put in the fields of Time Zone and Locale on the webpage. The result was log_timestamp “28/Apr/2014:18:23:21 +1000”, Graylog’s Timestamp “2018-11-08 17:56:30.795Z” and the timestamp field the same as the Graylog Timestamp “2018-11-08T17:56:30.795Z”
Again no success.
- I then found a comment by fobalili commented on Jan 11, 2016
However the grok pattern provided (a modification of COMMONAPACHELOG) raised an error. I assume this has something to do with the graylog version (I’m running the latest)
It would appear either it is not possible to do what I am trying to do or beyond my understanding/capability.
Can anyone tell me if it is possible and if so how (please dumb it down for me) or if it isn’t possible that would be useful as well.
Thanks,
Greg