1. Describe your incident:
Hello. Quite new graylog instalaltion on Oracle Linux, 3 graylogs in cluster . Running OK for more than 2 months. Then suddenly error when opening Alerts Window .
"Could not retrieve Event definition. …There was an error fetching a resource : IOException encountered while reading from a byte array input stream "
Similar error appears when clicking on Streams or Dashboars and clicking on Share icon:
"Preparing shares for AD failed: IOException encountered while reading from a byte array input stream
2. Describe your environment:
-
OS Information:
Oracle Linux 8.7 -
Package Version:
graylog RPM package 5.0.3+a82acb2, codename Noir
mongodb 5.0
opensearch 1.x (5 nodes)
3 graylog servers in a cluster -
Service logs, configurations, and environment variables:
Looking in graylog log it seems critical error to be connected to mongodb:
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "updated_at" (class org.graylog.events.processor.$AutoValue_EventDefinitionDto$Builder), not marked as ignorable (12 known properties: "field_spec", "config", "priority", "_scope", "notification_settings", "notifications", "_id", "storage", "description", "title", "key_spec", "alert"])
at [Source: de.undercouch.bson4jackson.io.LittleEndianInputStream@1ead1f3d; pos: 187] (through reference chain: org.graylog.events.processor.$AutoValue_EventDefinitionDto$Builder["updated_at"])
Longer version of that same error
2023-06-06T16:32:35.866+02:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:67) ~[graylog.jar:?]
at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:49) ~[graylog.jar:?]
at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:29) ~[graylog.jar:?]
at com.mongodb.internal.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52) ~[graylog.jar:?]
at com.mongodb.internal.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:60) ~[graylog.jar:
?]
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:87) ~[graylog.jar:?]
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:42) ~[graylog.jar:?]
at org.bson.internal.LazyCodec.decode(LazyCodec.java:48) ~[graylog.jar:?]
at org.bson.codecs.BsonDocumentCodec.readValue(BsonDocumentCodec.java:104) ~[graylog.jar:?]
at com.mongodb.internal.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:63) ~[graylog.jar:
?]
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:87) ~[graylog.jar:?]
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:42) ~[graylog.jar:?]
at com.mongodb.internal.connection.ReplyMessage.<init>(ReplyMessage.java:51) ~[graylog.jar:?]
at com.mongodb.internal.connection.InternalStreamConnection.getCommandResult(InternalStreamConnection.java:535) ~[graylog
.jar:?]
at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:4
20) ~[graylog.jar:?]
at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:337) ~[graylog.j
ar:?]
at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:116) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:643) ~[graylog.jar:?]
at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:71) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:240) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:226) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:126) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:116) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultServer$OperationCountTrackingConnection.command(DefaultServer.java:345) ~[graylog.jar:?]
at com.mongodb.internal.operation.CommandOperationHelper.createReadCommandAndExecute(CommandOperationHelper.java:232) ~[graylog.jar:?]
at com.mongodb.internal.operation.FindOperation.lambda$execute$1(FindOperation.java:695) ~[graylog.jar:?]
at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$2(OperationHelper.java:566) ~[graylog.jar:?]
at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:591) ~[graylog.jar:?]
at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$3(OperationHelper.java:565) ~[graylog.jar:?]
at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:591) ~[graylog.jar:?]
at com.mongodb.internal.operation.OperationHelper.withSourceAndConnection(OperationHelper.java:564) ~[graylog.jar:?]
at com.mongodb.internal.operation.FindOperation.lambda$execute$2(FindOperation.java:690) ~[graylog.jar:?]
at com.mongodb.internal.async.function.RetryingSyncSupplier.get(RetryingSyncSupplier.java:65) ~[graylog.jar:?]
at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:722) ~[graylog.jar:?]
at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:86) ~[graylog.jar:?]
at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:191) ~[graylog.jar:?]
at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:168) ~[graylog.jar:?]
at com.mongodb.DBCursor.initializeCursor(DBCursor.java:810) ~[graylog.jar:?]
at com.mongodb.DBCursor.hasNext(DBCursor.java:163) ~[graylog.jar:?]
at org.mongojack.DBCursor.hasNext(DBCursor.java:255) ~[graylog.jar:?]
at java.util.Iterator.forEachRemaining(Unknown Source) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.util.stream.ReferencePipeline.collect(Unknown Source) ~[?:?]
at org.graylog.events.contentpack.facade.EventDefinitionFacade.listEntityExcerpts(EventDefinitionFacade.java:177) ~[graylog.jar:?]
at org.graylog2.contentpacks.ContentPackService.lambda$listAllEntityExcerpts$3(ContentPackService.java:331) ~[graylog.jar:?]
at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422) ~[graylog.jar:?]
at org.graylog2.contentpacks.ContentPackService.listAllEntityExcerpts(ContentPackService.java:331) ~[graylog.jar:?]
at org.graylog.security.entities.EntityDependencyResolver.resolve(EntityDependencyResolver.java:69) ~[graylog.jar:?]
at org.graylog.security.shares.EntitySharesService.checkMissingPermissionsOnDependencies(EntitySharesService.java:307) ~[graylog.jar:?]
at org.graylog.security.shares.EntitySharesService.prepareShare(EntitySharesService.java:110) ~[graylog.jar:?]
at org.graylog.security.rest.EntitySharesResource.prepareShare(EntitySharesResource.java:124) ~[graylog.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "updated_at" (class org.graylog.events.processor.$AutoValue_EventDefinitionDto$Builder), not marked as ignorable (12 known properties: "field_spec", "config", "priority", "_scope", "notification_settings", "notifications", "_id", "storage", "description", "title", "key_spec", "alert"])
at [Source: de.undercouch.bson4jackson.io.LittleEndianInputStream@1ead1f3d; pos: 187] (through reference chain: org.graylog.events.processor.$AutoValue_EventDefinitionDto$Builder["updated_at"])
at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1127) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2036) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1700) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1678) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.vanillaDeserialize(BuilderBasedDeserializer.java:298) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:217) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4650) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2900) ~[graylog.jar:?]
at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:80) ~[graylog.jar:?]
at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:64) ~[graylog.jar:?]
... 81 more
core problem seem to be that GUI call is
3. What steps have you already taken to try and solve the problem?
I have just bit experience with doing minor config changes im mongodb, regarding adding nodes etc but this seems to be something deeper about event definition that is wrong? . For a test I loged into mongodb and started this command:
At least it didnt return error, just listed all events:
But in the output I could not find updated_at parameter
:PRIMARY> db.event_definitions.find()
{ "_id" : ObjectId("5d5bee50caa6e70dee6da620"), "title" : "Premalo msgov", "description" : "Migrated message count alert condition", "priority" : 2, "alert" : true, "config" : { "type" : "aggregation-v1", "query" : "*", "query_parameters" : [ ], "streams" : [ ], "group_by" : [ ], "series" : [ { "id" : "count-", "function" : "count", "field" : null } ], "conditions" : { "expression" : { "expr" : "<", "left" : { "expr" : "number-ref", "ref" : "count-" }, "right" : { "expr" : "number", "value" : 6000 } } }, "search_within_ms" : NumberLong(600000), "execute_every_ms" : NumberLong(600000) }, "field_spec" : { }, "key_spec" : [ ], "notification_settings" : { "grace_period_ms" : NumberLong(600000), "backlog_size" : NumberLong(0) }, "notifications" : [ { "notification_id" : "5d5bee50caa6e70dee6da61e", "notification_parameters" : null } ], "storage" : [ { "type" : "persist-to-streams-v1", "streams" : [ "000000000000000000000002" ] } ] }
{ "_id" : ObjectId("5d5bee50caa6e70dee6da626"), "title" : "domain test.org", "description" : "Migrated message count alert condition", "priority" : 2, "alert" : true, "config" : { "type" : "aggregation-v1", "query" : "domain:\"test.org\"", "query_parameters" : [ ], "streams" : [ "59f1c8a2fb1f6820b4c78bc4" ], "group_by" : [ ], "series" : [ { "id" : "count-", "function" : "count", "field" : null } ], "conditions" : { "expression" : { "expr" : ">", "left" : { "expr" : "number-ref", "ref" : "count-" }, "right" : { "expr" : "number", "value" : 0 } } }, "search_within_ms" : NumberLong(60000), "execute_every_ms" : NumberLong(60000) }, "field_spec" : { }, "key_spec" : [ ], "notification_settings" : { "grace_period_ms" : NumberLong(0), "backlog_size" : NumberLong(10) }, "notifications" : [ { "notification_id" : "5d65061e0714e852bdf72d6e", "notification_parameters" : null } ], "storage" : [ { "type" : "persist-to-streams-v1", "streams" : [ "000000000000000000000002" ] } ] }
..
I also read maybe similar case with mongo
Nullpointer Error on Alerts & Events AND Events Definition Page · Issue #8410 · Graylog2/graylog2-server · GitHub
4. How can the community help?
I guess solution is to find the right mongodb collection for this updated_at property and manipulate it ? But it is strange that GUI call is searching for field “updated_at”, and error saying that
“updated_at” (not marked as ignorable (12 known properties
Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]