Event Definitions cannot loading

Berk · March 26, 2020, 4:34pm

Hello,

We have graylog Graylog 3.1.4. We created two event definitions 2 months ago with email notificaitons, but now when i try to click event definitions it stucks at loading part like below;

When I checked the logs;

    2020-03-26T18:44:53.352+03:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
        at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:67) ~[graylog.jar:?]
        at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:49) ~[graylog.jar:?]
        at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:29) ~[graylog.jar:?]
        at com.mongodb.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52) ~[graylog.jar:?]
        at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:60) ~[graylog.jar:?]
        at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84) ~[graylog.jar:?]
        at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41) ~[graylog.jar:?]
        at org.bson.codecs.configuration.LazyCodec.decode(LazyCodec.java:47) ~[graylog.jar:?]
        at org.bson.codecs.BsonDocumentCodec.readValue(BsonDocumentCodec.java:101) ~[graylog.jar:?]
        at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:63) ~[graylog.jar:?]
        at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84) ~[graylog.jar:?]
        at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41) ~[graylog.jar:?]
        at com.mongodb.connection.ReplyMessage.<init>(ReplyMessage.java:51) ~[graylog.jar:?]
        at com.mongodb.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:301) ~[gray            log.jar:?]
        at com.mongodb.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:255) ~[graylog.jar:?]
        at com.mongodb.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:98) ~[graylo            g.jar:?]
        at com.mongodb.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:441) ~[graylog.            jar:?]
        at com.mongodb.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:80) ~[graylog.jar:?]
        at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:189) ~[graylog.jar:?]
        at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:264) ~[graylog.jar:?]
        at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:126) ~[graylog.jar:?]
        at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:118) ~[graylog.jar:?]
        at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:226) ~[graylog.j            ar:?]
        at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:217) ~[graylog.j            ar:?]
        at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:120) ~[graylog.j            ar:?]
        at com.mongodb.operation.FindOperation$1.call(FindOperation.java:717) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation$1.call(FindOperation.java:711) ~[graylog.jar:?]
        at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:471) ~[graylog.jar:?]
        at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:415) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation.execute(FindOperation.java:711) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation.execute(FindOperation.java:83) ~[graylog.jar:?]
        at com.mongodb.Mongo$3.execute(Mongo.java:826) ~[graylog.jar:?]
        at com.mongodb.Mongo$3.execute(Mongo.java:813) ~[graylog.jar:?]
        at com.mongodb.DBCursor.initializeCursor(DBCursor.java:877) ~[graylog.jar:?]
        at com.mongodb.DBCursor.hasNext(DBCursor.java:144) ~[graylog.jar:?]
        at org.mongojack.DBCursor.hasNext(DBCursor.java:330) ~[graylog.jar:?]
        at java.util.Iterator.forEachRemaining(Iterator.java:132) ~[?:?]
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:257) ~[?:?]
        at java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:248) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.count(ReferencePipeline.java:605) ~[?:?]
        at org.graylog2.database.PaginatedDbService.findPaginatedWithQueryFilterAndSort(PaginatedDbService.java:158) ~[graylog.jar            :?]
        at org.graylog.events.processor.DBEventDefinitionService.searchPaginated(DBEventDefinitionService.java:52) ~[graylog.jar:?            ]
        at org.graylog.events.rest.EventDefinitionsResource.list(EventDefinitionsResource.java:106) ~[graylog.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:567) ~[?:?]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHand            lerFactory.java:81) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatc            her.java:144) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispat            cher.java:161) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceM            ethodDispatcherProvider.java:205) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDisp            atcher.java:99) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) ~[graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) ~[graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) ~[graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) ~[grayl            og.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:830) ~[?:?]
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "query_parameters" (class org.gray            log.events.processor.aggregation.AutoValue_AggregationEventProcessorConfig$Builder), not marked as ignorable (8 known properties:             "streams", "execute_every_ms", "series", "query", "type", "search_within_ms", "group_by", "conditions"])
 at [Source: de.undercouch.bson4jackson.io.LittleEndianInputStream@11c2ad18; pos: 290] (through reference chain: org.graylog.event            s.processor.$AutoValue_EventDefinitionDto$Builder["config"]->org.graylog.events.processor.aggregation.AutoValue_AggregationEventPr            ocessorConfig$Builder["query_parameters"])
        at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:63) ~[graylog.            jar:?]
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:834) ~[graylog.            jar:?]
        at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1093) ~[graylog.jar            :?]
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1489) ~[grayl            og.jar:?]
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1467) ~[graylo            g.jar:?]
        at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserializeFromObject(BuilderBasedDeserializer.java:317)             ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:178) ~[graylog.            jar:?]
        at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializ            er.java:129) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeseri            alizer.java:97) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:209) ~[graylog.            jar:?]
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:502) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeSetAndReturn(MethodProperty.java:116) ~[graylog.jar            :?]
        at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.vanillaDeserialize(BuilderBasedDeserializer.java:266) ~[g            raylog.jar:?]
        at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:154) ~[graylog.            jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3786) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2181) ~[graylog.jar:?]
        at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:80) ~[graylog.jar:?]
        at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:64) ~[graylog.jar:?]
        ... 74 more

Do you have any recommendations to overcome this issue?

Ponet · March 26, 2020, 4:40pm

What has been changed between the initial creation of those event definitions and now?

Berk · March 26, 2020, 4:47pm

The elasticsearch indexes which are belongs to graylog events are re-created.

Also i can list and create new notifications without any problems.

Thank you for your quick reply.

Ponet · March 26, 2020, 4:58pm

Elasticsearch will have no effect on this.

Elasticsearch is only used for storing messages ingested by Graylog.

Event definitions, notifications and other Graylog components (dashboards, pipelines, etc) are stored in Mongodb.

Has Graylog been updated since the creation of those event definitions?

The one line from the error that caught my interest is:

Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "query_parameters" (class org.graylog.events.processor.aggregation.AutoValue_AggregationEventProcessorConfig$Builder), not marked as ignorable (8 known properties: "streams", "execute_every_ms", "series", "query", "type", "search_within_ms", "group_by", "conditions"])

Berk · March 26, 2020, 5:07pm

Recently, we have upgraded our servers from version 3.1.4 to 3.2.1 but, somehow, there were some unexpected problems with dashboards and sidecars. Therefore, we had to rollback our graylog to version 3.1.4.

Ponet · March 26, 2020, 5:13pm

That /might/ have caused this problem.

IIRC there were some changes made to the alerting/event system between 3.1 and 3.2 so, when you upgraded to 3.2 your event definitions will have been migrated to the new format.

Berk · March 26, 2020, 5:25pm

Thank you for your information. I think we should have made a backup of our mongodb before upgrading to 3.2.1, but i think unfortunately we did not make a backup. Is there any way to repeair it to the old format?

Ponet · March 26, 2020, 8:39pm

I’m not sure of specifics of the changes to be honest.

The easiest option might be to just delete the event definitions out of the mongodb which should at least resolve the loading issue and allow you to recreate the event definitions.

jan · March 29, 2020, 11:37am

@Berk

go to the latest bugfix release of Graylog - that should fix the problem.

Berk · April 5, 2020, 2:44pm

@Ponet thank you so much for your help. This solved our problem

system · April 19, 2020, 2:44pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
After updating graylog 4.0 to 5.1 unable to edit or create Event Definitions Graylog Central (peer support)	3	295	July 13, 2023
Fetching event definitions failed. Internal Server Error Graylog Central (peer support) components	7	3151	March 14, 2023
IOEXception when checking alerts Graylog Central (peer support)	5	198	January 29, 2024
Graylog Content Packs page will not load after upgrading from v3.0 to v3.1 Graylog Central (peer support)	4	1523	September 9, 2019
Graylog can not show event Definitions on web Graylog Central (peer support) alert	2	666	March 3, 2022

Event Definitions cannot loading

Related topics