Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!
1. Describe your incident:
unable to edit or create even definitions
2. Describe your environment:
-
OS Information:
Ubuntu 22.04 -
Package Version:
Graylog 5.1
mongodb 6.0.6
opensearch 1.3.10 -
Service logs, configurations, and environment variables:
2023-06-22T16:04:07.688Z ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:67) ~[graylog.jar:?]
at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:49) ~[graylog.jar:?]
at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:29) ~[graylog.jar:?]
at com.mongodb.internal.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52) ~[graylog.jar:?]
at com.mongodb.internal.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:60) ~[graylog.jar:?]
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:87) ~[graylog.jar:?]
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:42) ~[graylog.jar:?]
at org.bson.internal.LazyCodec.decode(LazyCodec.java:53) ~[graylog.jar:?]
at org.bson.codecs.BsonDocumentCodec.readValue(BsonDocumentCodec.java:104) ~[graylog.jar:?]
at com.mongodb.internal.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:63) ~[graylog.jar:?]
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:87) ~[graylog.jar:?]
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:42) ~[graylog.jar:?]
at com.mongodb.internal.connection.ReplyMessage.<init>(ReplyMessage.java:50) ~[graylog.jar:?]
at com.mongodb.internal.connection.InternalStreamConnection.getCommandResult(InternalStreamConnection.java:538) ~[graylog.jar:?]
at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:423) ~[graylog.jar:?]
at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:340) ~[graylog.jar:?]
at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:116) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:643) ~[graylog.jar:?]
at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:71) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:206) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:119) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:85) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:75) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultServer$OperationCountTrackingConnection.command(DefaultServer.java:293) ~[graylog.jar:?]
at com.mongodb.internal.operation.CommandOperationHelper.createReadCommandAndExecute(CommandOperationHelper.java:233) ~[graylog.jar:?]
at com.mongodb.internal.operation.FindOperation.lambda$execute$1(FindOperation.java:332) ~[graylog.jar:?]
at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$0(OperationHelper.java:356) ~[graylog.jar:?]
at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:381) ~[graylog.jar:?]
at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$1(OperationHelper.java:355) ~[graylog.jar:?]
at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:381) ~[graylog.jar:?]
at com.mongodb.internal.operation.OperationHelper.withSourceAndConnection(OperationHelper.java:354) ~[graylog.jar:?]
at com.mongodb.internal.operation.FindOperation.lambda$execute$2(FindOperation.java:329) ~[graylog.jar:?]
at com.mongodb.internal.async.function.RetryingSyncSupplier.get(RetryingSyncSupplier.java:67) ~[graylog.jar:?]
at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:340) ~[graylog.jar:?]
at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:79) ~[graylog.jar:?]
at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:191) ~[graylog.jar:?]
at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:168) ~[graylog.jar:?]
at com.mongodb.DBCursor.initializeCursor(DBCursor.java:810) ~[graylog.jar:?]
at com.mongodb.DBCursor.hasNext(DBCursor.java:163) ~[graylog.jar:?]
at org.mongojack.DBCursor.hasNext(DBCursor.java:255) ~[graylog.jar:?]
at java.util.Iterator.forEachRemaining(Unknown Source) ~[?:?]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source) ~[?:?]
at java.util.stream.ReduceOps$5.evaluateSequential(Unknown Source) ~[?:?]
at java.util.stream.ReduceOps$5.evaluateSequential(Unknown Source) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.util.stream.ReferencePipeline.count(Unknown Source) ~[?:?]
at org.graylog2.database.PaginatedDbService.findPaginatedWithQueryFilterAndSortWithGrandTotal(PaginatedDbService.java:200) ~[graylog.jar:?]
at org.graylog2.database.PaginatedDbService.findPaginatedWithQueryFilterAndSort(PaginatedDbService.java:181) ~[graylog.jar:?]
at org.graylog.events.notifications.DBNotificationService.searchPaginated(DBNotificationService.java:45) ~[graylog.jar:?]
at org.graylog.events.rest.EventNotificationsResource.listNotifications(EventNotificationsResource.java:164) ~[graylog.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "backlog_item_message" (class org.graylog.integrations.notifications.types.AutoValue_SlackEventNotificationConfig$Builder), not marked as ignorable (14 known properties: "icon_emoji", "include_title", "channel", "notify_here", "webhook_url", "icon_url", "notify_channel", "custom_message", "type", "time_zone", "user_name", "color", "backlog_size", "link_names"])
at [Source: de.undercouch.bson4jackson.io.LittleEndianInputStream@23fd5a78; pos: 375] (through reference chain: org.graylog.events.notifications.AutoValue_NotificationDto$Builder["config"]->org.graylog.integrations.notifications.types.AutoValue_SlackEventNotificationConfig$Builder["backlog_item_message"])
at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1127) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2036) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1700) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1678) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserializeFromObject(BuilderBasedDeserializer.java:344) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:240) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:144) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:110) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:263) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeSetAndReturn(MethodProperty.java:167) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.vanillaDeserialize(BuilderBasedDeserializer.java:293) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:217) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4650) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2900) ~[graylog.jar:?]
at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:80) ~[graylog.jar:?]
at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:64) ~[graylog.jar:?]
... 79 more
3. What steps have you already taken to try and solve the problem?
I update graylog from 4.0 to 5.1
I cloned the production server
upgraded that to elasticsearch 7.10.2 then exported the indexes
upgraded mongo from 4.0 to 4.2 then 4.2 to 4.4, then 4.4 to 5, then 5 to 6 and did a dump
created new server with
graylog5.1, opensearch 1.3.10, and mongodb 6.0.6
restored OS and Mongodb
copied existing server.conf and edited bind IP and URL to new server info
Started graylog and it starts fine
I am able to search the indexes as expected
but when I go to Alert Event Definitions, I am unable to view/edit or create new
I tried removing the Event Definitions in the gui and still unable to create
I tried removing them from the mogodb and restarting graylog and same thing.
can any one point me into the right direction?