Description of your problem
I have an event definition that executes every 24 hours to review the last 35 hours worth of logs for a specific message. This 24 hour rolling window is a hard requirement as the logs for the system being reviewed are only downlinked from a remote host once daily. I know best practice is 1:1 for “execute every”:“search within” but my use-case is a little atypical here and this is a requirement due to the downlinking of logs happening at random times throughout the morning each day. I think this is irrelevant to the actual issue at hand though.
I’m trying to better understand when this alert will fire off and how I can change it to better accommodate my team’s schedules. I’ve not been able to find a solid answer to this question yet in documentation/googling.
Description of steps you’ve taken to attempt to solve the issue
I have an event definition defined to review 35 hours worth of logs once daily:
There is no indicator of when I want this job to actually fire off anywhere on the event definition page, as best I can tell it seems to start the window when the event definition is created but I’d like to be able to set this time manually.
Reviewing the event_definitions entry in mongoDB, nothing I can find indicates what time this job fires off
{
"_id" : ObjectId("6112889c00a91a0537fb8ff3"),
"title" : "**ON-STATION** Ember Antenna Not Connected ",
"description" : "Generate Alerts when embers' emberd.log shows an antenna is not connected",
"priority" : 2,
"alert" : true,
"config" : {
"type" : "aggregation-v1",
"query" : "file: emberd AND message: (\"antenna\" AND \"not connected !!!!!\")",
"query_parameters" : [ ],
"streams" : [
"5ebaefc400a91a76558f0c14"
],
"group_by" : [ ],
"series" : [ ],
"conditions" : {
"expression" : null
},
"search_within_ms" : NumberLong(126000000),
"execute_every_ms" : NumberLong(86400000)
},
"field_spec" : {
},
"key_spec" : [ ],
"notification_settings" : {
"grace_period_ms" : NumberLong(30000),
"backlog_size" : NumberLong(50)
},
"notifications" : [
{
"notification_id" : "5f3fc8d300a91a48ddac1fbc",
"notification_parameters" : null
}
],
"storage" : [
{
"type" : "persist-to-streams-v1",
"streams" : [
"000000000000000000000002"
]
}
]
}
Likewise, reviewing the scheduler_job_definition from mongoDB is even more confusing. Not only does it not have the next time it’ll fire off listed, the definition seems to suggest this will only review an absolute, explicitly defined time window (which I know is incorrect, it’s definitely running every 24 hours).
db.scheduler_job_definitions.find( { title : {$regex : "Antenna"} } ).pretty()
{
"_id" : ObjectId("6112889c00a91a0537fb8ff4"),
"title" : "**ON-STATION** Ember Antenna Not Connected ",
"description" : "Generate Alerts when embers' emberd.log shows an antenna is not connected",
"config" : {
"type" : "event-processor-execution-v1",
"event_definition_id" : "6112889c00a91a0537fb8ff3",
"parameters" : {
"type" : "aggregation-v1",
"timerange" : {
"type" : "absolute",
"from" : ISODate("2021-08-09T03:41:01.448Z"),
"to" : ISODate("2021-08-10T14:41:01.448Z")
},
"streams" : [ ],
"batch_size" : 500
},
"processing_window_size" : NumberLong(126000000),
"processing_hop_size" : NumberLong(86400000)
}
}
Operating system information
- CentOS 7
Package versions
- Graylog – 3.2.6+e255fcc (from graylog-3.2-repository.noarch)
- MongoDB – v4.0.26
- Elasticsearch – 6.8.18 (see below)
{
"name" : "Wnw-8fG",
"cluster_name" : "graylog",
"cluster_uuid" : "dvFf5hhzTcSrOjw7Y4SeJQ",
"version" : {
"number" : "6.8.18",
"build_flavor" : "oss",
"build_type" : "rpm",
"build_hash" : "aca2329",
"build_date" : "2021-07-28T16:06:05.232873Z",
"build_snapshot" : false,
"lucene_version" : "7.7.3",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}