Field Content Alert Condition ignores logs which have log timestamp instead of indexed timestamp


#1

Graylog version=2.4.6

Hi, I am using “Field Content Alert Condition” as condition below, but my logs matching this condition does not fire the alert.
Field: level
Value: ERROR
Grace Period: 0
Message Backlog: 1
Repeat notifications: Checked

I did some investigation and found that “Field Content Alert Condition” searches logs having {CURRENT_TIME} - {ALERT_CHECK_INTERVAL} range timestamp.
(Currently my ${ALERT_CHECK_INTERVAL} is default value(=60s))
Because my log’s timestamp is not the time the log is indexed in Elasticsearch but the time when the log is generated actually,
those logs sometimes are not included in the range that Graylog alert engine checks.

e.g)
Log generated at 2018-12-01 09:00:00 (<- timestamp is this value)
Log indexed at 2018-12-01 09:05:00
Alert check starts at 2018-12-01 09:05:30

Are there any suggestions to solve this problem and make alert properly with log timestamp instead of indexed timestamp?