Event Definitions cannot loading

Graylog Central (peer support)
1

Hello,

We have graylog Graylog 3.1.4. We created two event definitions 2 months ago with email notificaitons, but now when i try to click event definitions it stucks at loading part like below;

image
image1903×243 21.9 KB

When I checked the logs;

    2020-03-26T18:44:53.352+03:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
        at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:67) ~[graylog.jar:?]
        at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:49) ~[graylog.jar:?]
        at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:29) ~[graylog.jar:?]
        at com.mongodb.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52) ~[graylog.jar:?]
        at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:60) ~[graylog.jar:?]
        at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84) ~[graylog.jar:?]
        at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41) ~[graylog.jar:?]
        at org.bson.codecs.configuration.LazyCodec.decode(LazyCodec.java:47) ~[graylog.jar:?]
        at org.bson.codecs.BsonDocumentCodec.readValue(BsonDocumentCodec.java:101) ~[graylog.jar:?]
        at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:63) ~[graylog.jar:?]
        at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84) ~[graylog.jar:?]
        at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41) ~[graylog.jar:?]
        at com.mongodb.connection.ReplyMessage.<init>(ReplyMessage.java:51) ~[graylog.jar:?]
        at com.mongodb.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:301) ~[gray            log.jar:?]
        at com.mongodb.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:255) ~[graylog.jar:?]
        at com.mongodb.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:98) ~[graylo            g.jar:?]
        at com.mongodb.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:441) ~[graylog.            jar:?]
        at com.mongodb.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:80) ~[graylog.jar:?]
        at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:189) ~[graylog.jar:?]
        at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:264) ~[graylog.jar:?]
        at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:126) ~[graylog.jar:?]
        at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:118) ~[graylog.jar:?]
        at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:226) ~[graylog.j            ar:?]
        at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:217) ~[graylog.j            ar:?]
        at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:120) ~[graylog.j            ar:?]
        at com.mongodb.operation.FindOperation$1.call(FindOperation.java:717) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation$1.call(FindOperation.java:711) ~[graylog.jar:?]
        at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:471) ~[graylog.jar:?]
        at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:415) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation.execute(FindOperation.java:711) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation.execute(FindOperation.java:83) ~[graylog.jar:?]
        at com.mongodb.Mongo$3.execute(Mongo.java:826) ~[graylog.jar:?]
        at com.mongodb.Mongo$3.execute(Mongo.java:813) ~[graylog.jar:?]
        at com.mongodb.DBCursor.initializeCursor(DBCursor.java:877) ~[graylog.jar:?]
        at com.mongodb.DBCursor.hasNext(DBCursor.java:144) ~[graylog.jar:?]
        at org.mongojack.DBCursor.hasNext(DBCursor.java:330) ~[graylog.jar:?]
        at java.util.Iterator.forEachRemaining(Iterator.java:132) ~[?:?]
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:257) ~[?:?]
        at java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:248) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.count(ReferencePipeline.java:605) ~[?:?]
        at org.graylog2.database.PaginatedDbService.findPaginatedWithQueryFilterAndSort(PaginatedDbService.java:158) ~[graylog.jar            :?]
        at org.graylog.events.processor.DBEventDefinitionService.searchPaginated(DBEventDefinitionService.java:52) ~[graylog.jar:?            ]
        at org.graylog.events.rest.EventDefinitionsResource.list(EventDefinitionsResource.java:106) ~[graylog.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:567) ~[?:?]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHand            lerFactory.java:81) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatc            her.java:144) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispat            cher.java:161) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceM            ethodDispatcherProvider.java:205) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDisp            atcher.java:99) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297) ~[graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267) ~[graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) ~[graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) ~[graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) ~[graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) ~[graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) ~[grayl            og.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:830) ~[?:?]
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "query_parameters" (class org.gray            log.events.processor.aggregation.AutoValue_AggregationEventProcessorConfig$Builder), not marked as ignorable (8 known properties:             "streams", "execute_every_ms", "series", "query", "type", "search_within_ms", "group_by", "conditions"])
 at [Source: de.undercouch.bson4jackson.io.LittleEndianInputStream@11c2ad18; pos: 290] (through reference chain: org.graylog.event            s.processor.$AutoValue_EventDefinitionDto$Builder["config"]->org.graylog.events.processor.aggregation.AutoValue_AggregationEventPr            ocessorConfig$Builder["query_parameters"])
        at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:63) ~[graylog.            jar:?]
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:834) ~[graylog.            jar:?]
        at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1093) ~[graylog.jar            :?]
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1489) ~[grayl            og.jar:?]
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1467) ~[graylo            g.jar:?]
        at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserializeFromObject(BuilderBasedDeserializer.java:317)             ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:178) ~[graylog.            jar:?]
        at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializ            er.java:129) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeseri            alizer.java:97) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:209) ~[graylog.            jar:?]
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:502) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeSetAndReturn(MethodProperty.java:116) ~[graylog.jar            :?]
        at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.vanillaDeserialize(BuilderBasedDeserializer.java:266) ~[g            raylog.jar:?]
        at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:154) ~[graylog.            jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3786) ~[graylog.jar:?]
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2181) ~[graylog.jar:?]
        at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:80) ~[graylog.jar:?]
        at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:64) ~[graylog.jar:?]
        ... 74 more

Do you have any recommendations to overcome this issue?

2

What has been changed between the initial creation of those event definitions and now?

3

The elasticsearch indexes which are belongs to graylog events are re-created.

Also i can list and create new notifications without any problems.

Thank you for your quick reply.

4

Elasticsearch will have no effect on this.

Elasticsearch is only used for storing messages ingested by Graylog.

Event definitions, notifications and other Graylog components (dashboards, pipelines, etc) are stored in Mongodb.

Has Graylog been updated since the creation of those event definitions?

The one line from the error that caught my interest is:

Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "query_parameters" (class org.graylog.events.processor.aggregation.AutoValue_AggregationEventProcessorConfig$Builder), not marked as ignorable (8 known properties: "streams", "execute_every_ms", "series", "query", "type", "search_within_ms", "group_by", "conditions"])
5

Recently, we have upgraded our servers from version 3.1.4 to 3.2.1 but, somehow, there were some unexpected problems with dashboards and sidecars. Therefore, we had to rollback our graylog to version 3.1.4.

6

That /might/ have caused this problem.

IIRC there were some changes made to the alerting/event system between 3.1 and 3.2 so, when you upgraded to 3.2 your event definitions will have been migrated to the new format.

1 Like
7

Thank you for your information. I think we should have made a backup of our mongodb before upgrading to 3.2.1, but i think unfortunately we did not make a backup. Is there any way to repeair it to the old format?

8

I’m not sure of specifics of the changes to be honest.

The easiest option might be to just delete the event definitions out of the mongodb which should at least resolve the loading issue and allow you to recreate the event definitions.

1 Like
9

@Berk

go to the latest bugfix release of Graylog - that should fix the problem.

1 Like
10

@Ponet thank you so much for your help. This solved our problem :slight_smile:

1 Like
11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.