Event Failures after upgrade from 7.0.6 to 7.1.1

1. Describe your incident

Since migrating from Graylog 7.0.6 to 7.1.0, the Event Definitions page (/alerts/definitions) no longer loads any content. The UI displays the following error at the bottom of the page:

Could not retrieve event definitions
Fetching event definitions failed with status: FetchError: There was an error fetching a resource: . Additional information: IOException encountered while parsing

The “Create Event Definition” button is visible and accessible, but submitting the creation form triggers the same error. No event definitions are displayed, making it impossible to view or manage existing alert configurations.

This issue appeared immediately after the upgrade from 7.0.6 to 7.1.0 and persists after a second upgrade to 7.1.1. The problem is fully reproducible.


2. Describe your environment

OS Information:

  • Debian 12.13 (Bookworm) — x86_64

Package Version:

  • Graylog: 7.1.1-1 (upgraded from 7.0.6 → 7.1.0 → 7.1.1)
  • MongoDB: 7.0.32
  • OpenSearch: 2.19.5
  • Java: 21.0.9 LTS

3. What steps have you already taken to try and solve the problem?

  • Restarted the Graylog service after the migration
  • Verified that MongoDB and OpenSearch services are running and healthy
  • Confirmed the issue is reproducible across multiple browsers (ruling out a cache problem)
  • Checked Graylog server logs for IOException errors related to event definition parsing
  • Upgraded a second time from 7.1.0 to 7.1.1 hoping it would resolve the issue — problem persists
  • Reviewed the Graylog 7.1 upgrade documentation — no specific breaking change mentioned for event definitions

4. How can the community help?

  • Is there a known breaking change or migration step for Event Definitions between 7.0.6 and 7.1.x that could cause an IOException during parsing?
  • Could this be a MongoDB schema change that requires a manual migration script?
  • Is there a way to re-index or repair the event definitions collection without losing existing configurations?
  • Any known workaround to restore access to the Event Definitions UI while investigating the root cause?

Hey @lpop,

I’ve not yet seen this specific issue with the update to 7.1.1, does the error within the log file have a stack trace you could post?

It’s fine, I found the cause, it was coming from an alert I had created with the plugin https://github.com/airbus-cyber/graylog-plugin-correlation-count which seemed to be causing an issue. I deleted it and now everything is working fine.

we upgraded 7.1.0 to 7.1.1 and experienced the same problem. We do not have any extra plugins installed. Downgraded to 7.1.0 and is fine again.

Hey @rodney_vdw,

Is there a stack trace produced within the server.log (usually found /var/log/graylog-server/server.log) when this error is thrown?

Only see this

2026-05-13T12:12:50.975Z ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.RuntimeException: Unable to perform search query
	at org.graylog.storage.opensearch3.OfficialOpensearchClient.mapException(OfficialOpensearchClient.java:207)
	at org.graylog.storage.opensearch3.OfficialOpensearchClient.sync(OfficialOpensearchClient.java:81)
	at org.graylog.storage.opensearch3.MoreSearchAdapterOS.eventSearch(MoreSearchAdapterOS.java:131)
	at org.graylog.events.search.MoreSearch.eventSearch(MoreSearch.java:107)
	at org.graylog.events.search.EventsSearchService.search(EventsSearchService.java:60)
	at org.graylog.events.search.EventsSearchService.searchByIds(EventsSearchService.java:111)
	at org.graylog.events.rest.EventsResource.getById(EventsResource.java:108)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:470)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:394)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:266)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:253)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:374)
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190)
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:259)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [shard_failure] 1 of 5 shards failed
	at org.graylog.storage.opensearch3.client.CustomOpenSearchClient.throwOnShardFailures(CustomOpenSearchClient.java:73)
	at org.graylog.storage.opensearch3.client.CustomOpenSearchClient.search(CustomOpenSearchClient.java:61)
	at org.graylog.storage.opensearch3.MoreSearchAdapterOS.lambda$eventSearch$2(MoreSearchAdapterOS.java:131)
	at org.graylog.storage.opensearch3.OfficialOpensearchClient.sync(OfficialOpensearchClient.java:79)
	... 30 more

for info we run an opensearch 2.19.5 external cluster (not data nodes)

It seems like this is occurring due to a failing shard, can you get cat shards on your Opensearch cluster to see if there are any failing shards?

apologies for the delay @Wine_Merchant

i redid the upgrade today and again the same problem. opensearch cluster reports green, nothing in error logs, only the java exception in graylog which was shared previously.. rolled back to 7.0.8 and all is well once again. I guess if this is not a bigger issue with other users upgrading then perhaps something local to our setup - but dont have the to debug now.

Morning @rodney_vdw,

This issue is actually down to change made to how errors thrown due to failing shards are surfaced in the product. The change now explicitly checks for partial shard failures and throws rather than returning incomplete results wheres before it would have returned results.

There will hopefully be be a change made in 7.1.3 to adrress what you are experiencing.

can confirm (at least at a glance) that upgrading to 7.1.3 has resolved the issue.. thanks for the input.