Couldn't find job definition

[tywjohn (Ty John)]

Getting the following error message repeat itself several times a second after upgrading from 3.2.2 to 3.2.3. I don’t even know where to begin looking for the source of the issue. Any ideas?

Also mongodb is sitting at almost 100% CPU. Pretty sure that’s not normal.

2020-03-18T17:13:12.796+10:30 ERROR [JobExecutionEngine] Couldn't handle trigger due to a permanent error 5e713508666462083e52b903 - trigger won't be retried
java.lang.IllegalStateException: Couldn't find job definition 5e6afbb5666462083e898539
        at org.graylog.scheduler.JobExecutionEngine.lambda$handleTrigger$1(JobExecutionEngine.java:137) ~[graylog.jar:?]
        at java.util.Optional.orElseThrow(Optional.java:290) ~[?:1.8.0_242]
        at org.graylog.scheduler.JobExecutionEngine.handleTrigger(JobExecutionEngine.java:137) ~[graylog.jar:?]
        at org.graylog.scheduler.JobExecutionEngine.lambda$execute$0(JobExecutionEngine.java:119) ~[graylog.jar:?]
        at org.graylog.scheduler.worker.JobWorkerPool.lambda$execute$0(JobWorkerPool.java:110) ~[graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]

[tywjohn (Ty John)]

OK some more information. I ran ‘echo ‘db.scheduler_triggers.find()’ | mongo graylog’ and I get several entries similar to below.
It looks like what has happened is I created an event trigger with an email alert but I didn’t have email smtp settings configured so it wasn’t able to send. I upgraded to 3.2.3 and deleted the event and alert objects but it seems there are some remnants in the database. Does anyone know how I can remove them?

MongoDB shell version v4.0.16
connecting to: mongodb://127.0.0.1:27017/graylog?gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("2e1f767f-fe8c-4de4-9685-5518c316a57c") }
MongoDB server version: 4.0.16
{ "_id" : ObjectId("5e6afbde666462083e89856f"), "job_definition_id" : "5e6afbb5666462083e898539", "start_time" : ISODate("2020-03-13T03:19:58.160Z"), "end_time" : null, "next_time" : ISODate("2020-03-18T05:01:37.878Z"), "created                                                                                           _at" : ISODate("2020-03-13T03:19:58.160Z"), "updated_at" : ISODate("2020-03-13T03:19:58.160Z"), "triggered_at" : ISODate("2020-03-18T04:56:37.803Z"), "status" : "runnable", "lock" : { "clock" : NumberLong(0), "progress" : 0, "la                                                                                           st_lock_time" : ISODate("2020-03-18T04:56:37.803Z"), "owner" : null }, "schedule" : { "type" : "once" }, "data" : { "type" : "notification-execution-v1", "event_dto" : { "id" : "01E38YFPXH8GXX314P6MCZTMX1", "event_definition_typ                                                                                           e" : "aggregation-v1", "event_definition_id" : "5e6afbd7666462083e898564", "origin_context" : "urn:graylog:message:es:graylog_5:59a21990-6423-11ea-9a6e-005056aa67c2", "timestamp" : ISODate("2020-03-12T05:35:40Z"), "timestamp_pro                                                                                           cessing" : ISODate("2020-03-13T03:19:58.129Z"), "timerange_start" : null, "timerange_end" : null, "streams" : [ ], "source_streams" : [ "000000000000000000000001" ], "message" : "Threat Detected", "source" : "redacted", "key_tuple" : [ ], "key" : "", "priority" : NumberLong(2), "alert" : true, "fields" : {  } } } }

[tywjohn (Ty John)]

For anyone stumbling into this issue, I figured out how to delete objects from mongodb and just removed everything from the scheduled_triggers collection which has resolved the issue.

sudo -u graylog echo 'db.scheduler_triggers.remove({})' | mongo graylog