I have upgraded Graylog from 3.3.10 to 4.0.2 and at first things seemed to work just fine.
Soon enough I noticed that the Streams tab doesn’t load anymore, it throws an error 500.
I have seen at least one other topic here with the exact same problem, but there was no solution provided when that topic got closed automatically.
I’ve downgraded back to 3.3.10 and everything works again, but I’d like to be able to use the newer version.
So, I’m creating this new topic and hope that someone can help me fix this issue.
Some version info:
Graylog 3.3.10
Elasticsearch 6.8.13
MongoDB 4.2.12
Java: OpenJDK 11.0.9.1
Using the API browser, it throws an error for the paginated list of streams:
"type": "ApiError",
"message": "IOException encountered while reading from a byte array input stream"
As soon as I access the paginated version of my streams, this shows up in the error log:
2021-02-08T15:55:02.323+01:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource java.lang.RuntimeException: IOException encountered while reading from a byte array input stream at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:67) ~[graylog.jar:?] at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:49) ~[graylog.jar:?] at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:29) ~[graylog.jar:?] at com.mongodb.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52) ~[graylog.jar:?] at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:60) ~[graylog.jar:?] at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84) ~[graylog.jar:?] at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41) ~[graylog.jar:?] at org.bson.internal.LazyCodec.decode(LazyCodec.java:48) ~[graylog.jar:?] at org.bson.codecs.BsonDocumentCodec.readValue(BsonDocumentCodec.java:101) ~[graylog.jar:?] at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:63) ~[graylog.jar:?] at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84) ~[graylog.jar:?] at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41) ~[graylog.jar:?] at com.mongodb.internal.connection.ReplyMessage.<init>(ReplyMessage.java:51) ~[graylog.jar:?] at com.mongodb.internal.connection.InternalStreamConnection.getCommandResult(InternalStreamConnection.java:413) ~[graylog.jar:?] at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:309) ~[graylog.jar:?] at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:259) ~[graylog.jar:?] at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99) ~[graylog.jar:?] at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:450) ~[graylog.jar:?] at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72) ~[graylog.jar:?] at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:226) ~[graylog.jar:?] at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269) ~[graylog.jar:?] at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131) ~[graylog.jar:?] at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123) ~[graylog.jar:?] at com.mongodb.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:343) ~[graylog.jar:?] at com.mongodb.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:334) ~[graylog.jar:?] at com.mongodb.operation.CommandOperationHelper.executeCommandWithConnection(CommandOperationHelper.java:220) ~[graylog.jar:?] at com.mongodb.operation.FindOperation$1.call(FindOperation.java:731) ~[graylog.jar:?] at com.mongodb.operation.FindOperation$1.call(FindOperation.java:725) ~[graylog.jar:?] at com.mongodb.operation.OperationHelper.withReadConnectionSource(OperationHelper.java:463) ~[graylog.jar:?] at com.mongodb.operation.FindOperation.execute(FindOperation.java:725) ~[graylog.jar:?] at com.mongodb.operation.FindOperation.execute(FindOperation.java:89) ~[graylog.jar:?] at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:196) ~[graylog.jar:?] at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:177) ~[graylog.jar:?] at com.mongodb.DBCursor.initializeCursor(DBCursor.java:989) ~[graylog.jar:?] at com.mongodb.DBCursor.hasNext(DBCursor.java:172) ~[graylog.jar:?] at org.mongojack.DBCursor.hasNext(DBCursor.java:330) ~[graylog.jar:?] at java.util.Iterator.forEachRemaining(Iterator.java:132) ~[?:?] at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?] at java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:257) ~[?:?] at java.util.stream.ReduceOps$5.evaluateSequential(ReduceOps.java:248) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?] at java.util.stream.ReferencePipeline.count(ReferencePipeline.java:605) ~[?:?] at org.graylog2.database.PaginatedDbService.findPaginatedWithQueryFilterAndSort(PaginatedDbService.java:159) ~[graylog.jar:?] at org.graylog2.streams.PaginatedStreamService.findPaginated(PaginatedStreamService.java:48) ~[graylog.jar:?] at org.graylog2.rest.resources.streams.StreamResource.getPage(StreamResource.java:210) ~[graylog.jar:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[graylog.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[graylog.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[graylog.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[graylog.jar:?] at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) [graylog.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?] at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) [graylog.jar:?] at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) [graylog.jar:?] at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?] at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?] at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `org.graylog2.streams.StreamDTO`, problem: Null matchingType at [Source: de.undercouch.bson4jackson.io.LittleEndianInputStream@3f033396; pos: 536] at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67) ~[graylog.jar:?] at com.fasterxml.jackson.databind.DeserializationContext.instantiationException(DeserializationContext.java:1608) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapAsJsonMappingException(StdValueInstantiator.java:484) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.rewrapCtorProblem(StdValueInstantiator.java:503) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:285) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith(ValueInstantiator.java:229) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:195) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:488) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1287) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159) ~[graylog.jar:?] at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3985) ~[graylog.jar:?] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2343) ~[graylog.jar:?] at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:80) ~[graylog.jar:?] at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:64) ~[graylog.jar:?] ... 74 more Caused by: java.lang.NullPointerException: Null matchingType at org.graylog2.streams.$AutoValue_StreamDTO.<init>($AutoValue_StreamDTO.java:71) ~[graylog.jar:?] at org.graylog2.streams.AutoValue_StreamDTO.<init>(AutoValue_StreamDTO.java:19) ~[graylog.jar:?] at org.graylog2.streams.StreamDTO.create(StreamDTO.java:141) ~[graylog.jar:?] at jdk.internal.reflect.GeneratedMethodAccessor293.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at com.fasterxml.jackson.databind.introspect.AnnotatedMethod.call(AnnotatedMethod.java:109) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:283) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith(ValueInstantiator.java:229) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:195) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:488) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1287) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159) ~[graylog.jar:?] at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3985) ~[graylog.jar:?] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2343) ~[graylog.jar:?] at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:80) ~[graylog.jar:?] at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:64) ~[graylog.jar:?] ... 74 more
I have tried removing all my own streams, but that didn’t help.
Dropping the streams collection in MongoDB didn’t help either.
I’m out of ideas here, now I’m hoping that someone has a solution!
Thanks for reading.
Bob