1. Describe your incident:
Twice in the past couple of weeks I have observed a situation where an event definition should generate an alert and corresponding notification but does not. I can take the event definition filter parameters back to the pertinent stream and see that they match, but no alert was triggered. I can recreate the situation in which an alert should be triggered by the event definition, observe the qualifying message populate the stream within the defined search window, and no alert is triggered. If I disable the event definition and re-enable it the event definition will then begin triggering alerts as expected.
Looking at the mongo tables I see that the event definition “alert” field is set to true in the event_definitions collection. I see that the “max_processed_timestamp” field is recent (in line with other event definitions) in the event_processor_state collection.
Has anyone else encountered this?
2. Describe your environment:
- OS Information:
- Package Version:
- Service logs, configurations, and environment variables:
3. What steps have you already taken to try and solve the problem?
Looked for obvious misconfigurations in the mongo collections. Resetting event definitions corrects it, but doesn’t inspire much confidence.
4. How can the community help?
Hoping someone else has run into and solved this already.