I’ve read through a number of threads/topics/solutions both in the community and general google but I still can’t get a working solution.
I have been unable to replace the graylog ingestion timestamp with the log message’s timestamp.
I have tried using
- an extractor using COMMONAPACHELOG with the HTTPDATE going to timestamp field. This had no impact the timestamp was equal to the Timestamp (log ingestion time).
- modified extractor by expanding the COMMONAPACHELOG grok pattern and writing HTTPDATE to log_timestamp which showed the correct value in log_timestamp as “28/Apr/2014:18:23:21 +1000”.
- I then added a pipeline with the pipeline rule
rule “substitute ingest time to log_time”
let new_time = parse_date(to_string($message.log_timestamp),“dd/MM/yyyy:HH:mm:ss”);
This did not change the timestamp field
- I then disconnected the pipeline and tried adding a second extractor to the COMMONAPACHELOG with the field going to log_timestamp. This second extractor set up with “source field” of log_timestamp the “store as field” set to timestamp, a converter of “Convert date type” with format string dd-MMM-yyyy:HH:mm:ss Z with no value put in the fields of Time Zone and Locale on the webpage. The result was log_timestamp “28/Apr/2014:18:23:21 +1000”, Graylog’s Timestamp “2018-11-08 17:56:30.795Z” and the timestamp field the same as the Graylog Timestamp “2018-11-08T17:56:30.795Z”
Again no success.
- I then found a comment by fobalili commented on Jan 11, 2016
However the grok pattern provided (a modification of COMMONAPACHELOG) raised an error. I assume this has something to do with the graylog version (I’m running the latest)
It would appear either it is not possible to do what I am trying to do or beyond my understanding/capability.
Can anyone tell me if it is possible and if so how (please dumb it down for me) or if it isn’t possible that would be useful as well.