Sorry for my late reply, I haven’t had any time to focus on this problem and unfortunately my original topic has already closed automatically.
I’d like to continue here and answer the questions that were asked in my original topic here.
I’d like to also thank you guys for trying to help me out!
I now have 2 instances of Graylog running, because I copied the existing installation and content to a new Debian server. Graylog on this new server has been downgraded to 3.3.10, because that is a working version for me.
I can test on the old server, which I have just upgraded to 4.0.5. Unfortunately, I still have the same issues.
On this instance Graylog is running on Java 1.8.0:
~# ps waux | grep -i graylog graylog 14006 0.0 0.0 2388 696 ? Ss 10:23 0:00 /bin/sh /usr/share/graylog-server/bin/graylog-server graylog 14029 27.2 12.3 4817756 1011888 ? Sl 10:23 1:23 /usr/lib/jvm/java-8-oracle/jre/bin/java -Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Duser.timezone=Europe/Amsterdam -XX:+UseParNewGC -jar -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb /usr/share/graylog-server/graylog.jar server -f /etc/graylog/server/server.conf -np
And here is a dump of all streams by issuing db.streams.find() in Mongo:
{ "_id" : ObjectId("000000000000000000000001"), "creator_user_id" : "local:admin", "is_default_stream" : true, "index_set_id" : "58e4e9a5407a8c7ac2bf92f4", "matching_type" : "AND", "remove_matches_from_default_stream" : false, "description" : "Stream containing all messages", "created_at" : ISODate("2017-04-05T12:57:09.683Z"), "disabled" : false, "title" : "All messages" }
{ "_id" : ObjectId("531870bfe4b09096d0318777"), "alert_conditions" : [ { "creator_user_id" : "bob", "created_at" : "2014-10-07T11:58:18.924Z", "id" : "9484c7ba-4ee6-48fd-9a4f-83371dcc31df", "type" : "message_count", "title" : "User enabled or disabled", "parameters" : { "backlog" : 1, "repeat_notifications" : false, "grace" : 0, "threshold_type" : "MORE", "threshold" : 0, "time" : 2 } } ], "created_at" : ISODate("2014-03-06T12:57:35.647Z"), "creator_user_id" : "bob", "description" : "User account disabled or enabled", "disabled" : false, "index_set_id" : "58e4e9a5407a8c7ac2bf92f4", "title" : "User account was enabled or disabled" }
{ "_id" : ObjectId("56e7e17de4b027b074a0ca11"), "alert_conditions" : [ { "creator_user_id" : "bob", "created_at" : "2016-03-15T10:24:01.751Z", "id" : "da0c704e-2982-4546-be9c-db3c967a07c4", "type" : "message_count", "title" : "Locked out users", "parameters" : { "backlog" : 5, "repeat_notifications" : false, "grace" : 15, "threshold_type" : "MORE", "threshold" : 5, "time" : 2 } } ], "content_pack" : null, "created_at" : ISODate("2016-03-15T10:18:37.878Z"), "creator_user_id" : "bob", "description" : "Locked out users", "disabled" : false, "index_set_id" : "58e4e9a5407a8c7ac2bf92f4", "matching_type" : "AND", "title" : "locked out users" }
{ "_id" : ObjectId("5ad5d007407a8c1fbe10f1d5"), "creator_user_id" : "bob", "index_set_id" : "58e4e9a5407a8c7ac2bf92f4", "matching_type" : "AND", "remove_matches_from_default_stream" : false, "created_at" : ISODate("2018-04-17T10:44:23.047Z"), "description" : "User added to/removed from Enterprise Admins group", "disabled" : false, "alert_conditions" : [ { "creator_user_id" : "bob", "created_at" : "2018-04-17T10:44:23.052Z", "id" : "d2ac9055-87d4-430f-98f8-022d980c08f7", "type" : "message_count", "title" : "(A)Enterprise Admin user added or removed", "parameters" : { "backlog" : 5, "repeat_notifications" : false, "query" : "*", "grace" : 0, "threshold_type" : "MORE", "threshold" : 0, "time" : 2 } } ], "title" : "Enterprise Admins" }
{ "_id" : ObjectId("5ad5d1ae407a8c1fbe10f3c3"), "creator_user_id" : "bob", "index_set_id" : "58e4e9a5407a8c7ac2bf92f4", "matching_type" : "AND", "remove_matches_from_default_stream" : false, "created_at" : ISODate("2018-04-17T10:51:26.625Z"), "description" : "User added to/removed from Schema Admins group", "disabled" : false, "alert_conditions" : [ { "creator_user_id" : "bob", "created_at" : "2018-04-17T10:51:26.629Z", "id" : "fdfab9ce-b3e4-4522-8c00-1cf4e6dea616", "type" : "message_count", "title" : "(A)Schema Admin user added or removed", "parameters" : { "backlog" : 5, "repeat_notifications" : false, "query" : "*", "grace" : 0, "threshold_type" : "MORE", "threshold" : 0, "time" : 2 } } ], "title" : "Schema Admins" }
{ "_id" : ObjectId("5c90b03e407a8c4f2474d609"), "creator_user_id" : "bob", "index_set_id" : "58e4e9a5407a8c7ac2bf92f4", "matching_type" : "AND", "remove_matches_from_default_stream" : false, "created_at" : ISODate("2019-03-19T09:02:54.663Z"), "description" : "User added to Organization Management group", "disabled" : false, "alert_conditions" : [ { "creator_user_id" : "bob", "created_at" : "2019-03-19T09:32:07.748Z", "id" : "6d2d976e-4779-44a0-b130-ff72fafdd226", "type" : "message_count", "title" : "(E)User added to or removed from \"Organization Management\"", "parameters" : { "backlog" : 1, "repeat_notifications" : false, "grace" : 0, "threshold_type" : "MORE", "threshold" : 0, "time" : 2 } } ], "title" : "Organization Management (Exchange)", "content_pack" : null }
{ "_id" : ObjectId("5c90b575407a8c4f2474dbc7"), "alert_conditions" : [ { "creator_user_id" : "bob", "created_at" : "2019-03-19T09:37:40.540Z", "id" : "d18997bc-be1c-4043-87b5-df79efb1e02d", "type" : "message_count", "title" : "(E)User added to or removed from \"Exchange Organization Administrators\"", "parameters" : { "backlog" : 1, "repeat_notifications" : false, "grace" : 0, "threshold_type" : "MORE", "threshold" : 0, "time" : 2 } } ], "created_at" : ISODate("2019-03-19T09:25:09.615Z"), "creator_user_id" : "bob", "description" : "User added to/removed from Exchange Organization Administrators group", "disabled" : false, "index_set_id" : "58e4e9a5407a8c7ac2bf92f4", "matching_type" : "AND", "remove_matches_from_default_stream" : false, "title" : "Exchange Organization Administrators" }
{ "_id" : ObjectId("5e32d2c0407a8c277f55df85"), "creator_user_id" : "bob", "index_set_id" : "58e4e9a5407a8c7ac2bf92f4", "matching_type" : "AND", "remove_matches_from_default_stream" : false, "description" : "Events", "created_at" : ISODate("2020-01-30T12:57:36.499Z"), "disabled" : false, "title" : "Insecure LDAP Bindings", "content_pack" : null }
{ "_id" : ObjectId("000000000000000000000002"), "creator_user_id" : "admin", "is_default_stream" : false, "index_set_id" : "5e172e40407a8c5fe3d3165c", "matching_type" : "AND", "remove_matches_from_default_stream" : true, "description" : "Stream containing all events created by Graylog", "created_at" : ISODate("2021-02-08T09:11:47.442Z"), "disabled" : false, "title" : "All events" }
{ "_id" : ObjectId("000000000000000000000003"), "creator_user_id" : "admin", "is_default_stream" : false, "index_set_id" : "5e172e40407a8c5fe3d3165e", "matching_type" : "AND", "remove_matches_from_default_stream" : true, "description" : "Stream containing all system events created by Graylog", "created_at" : ISODate("2021-02-08T09:11:47.463Z"), "disabled" : false, "title" : "All system events" }
{ "_id" : ObjectId("60211c4a5dc5b85ad1a0c5b5"), "creator_user_id" : "bob", "index_set_id" : "58e4e9a5407a8c7ac2bf92f4", "matching_type" : "AND", "remove_matches_from_default_stream" : false, "description" : "User added to/removed from Domain Admins group", "created_at" : ISODate("2021-02-08T11:11:06.254Z"), "disabled" : false, "title" : "Domain Admins", "content_pack" : null }
When using the API browser, I can replicate the error by using the “GET /streams/paginated” operation. All the other operations seem to work fin (that includes “GET /streams”).
The response body for “GET /streams/paginated” is:
{ "type": "ApiError", "message": "IOException encountered while reading from a byte array input stream" }
And the response header:
{"Connection":"close","Content-Length":"100","Content-Type":"application/json","Date":"Wed, 03 Mar 2021 09:59:06 GMT","Server":"Apache/2.4.38","X-Graylog-Node-Id":"4e0cdc14-73e3-41f5-a907-45125e6c8cb6","X-Runtime-Microseconds":"4884"}
For even more error information, here is the graylog-server log when this happens:
2021-03-03T11:06:21.790+01:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource java.lang.RuntimeException: IOException encountered while reading from a byte array input stream at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:67) ~[graylog.jar:?] at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:49) ~[graylog.jar:?] at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:29) ~[graylog.jar:?] at com.mongodb.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52) ~[graylog.jar:?] at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:60) ~[graylog.jar:?] at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84) ~[graylog.jar:?] at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41) ~[graylog.jar:?] at org.bson.internal.LazyCodec.decode(LazyCodec.java:48) ~[graylog.jar:?] at org.bson.codecs.BsonDocumentCodec.readValue(BsonDocumentCodec.java:101) ~[graylog.jar:?] at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:63) ~[graylog.jar:?] at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84) ~[graylog.jar:?] at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41) ~[graylog.jar:?] at com.mongodb.internal.connection.ReplyMessage.<init>(ReplyMessage.java:51) ~[graylog.jar:?] at com.mongodb.internal.connection.InternalStreamConnection.getCommandResult(InternalStreamConnection.java:413) ~[graylog.jar:?] at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:309) ~[graylog.jar:?] at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:259) ~[graylog.jar:?] at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99) ~[graylog.jar:?] at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:450) ~[graylog.jar:?] at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72) ~[graylog.jar:?] at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:226) ~[graylog.jar:?] at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269) ~[graylog.jar:?] at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131) ~[graylog.jar:?] at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123) ~[graylog.jar:?] at com.mongodb.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:343) ~[graylog.jar:?] at com.mongodb.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:334) ~[graylog.jar:?] at com.mongodb.operation.CommandOperationHelper.executeCommandWithConnection(CommandOperationHelper.java:220) ~[graylog.jar:?] at com.mongodb.operation.FindOperation$1.call(FindOperation.java:731) ~[graylog.jar:?] at com.mongodb.operation.FindOperation$1.call(FindOperation.java:725) ~[graylog.jar:?] at com.mongodb.operation.OperationHelper.withReadConnectionSource(OperationHelper.java:463) ~[graylog.jar:?] at com.mongodb.operation.FindOperation.execute(FindOperation.java:725) ~[graylog.jar:?] at com.mongodb.operation.FindOperation.execute(FindOperation.java:89) ~[graylog.jar:?] at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:196) ~[graylog.jar:?] at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:177) ~[graylog.jar:?] at com.mongodb.DBCursor.initializeCursor(DBCursor.java:989) ~[graylog.jar:?] at com.mongodb.DBCursor.hasNext(DBCursor.java:172) ~[graylog.jar:?] at org.mongojack.DBCursor.hasNext(DBCursor.java:330) ~[graylog.jar:?] at java.util.Iterator.forEachRemaining(Iterator.java:115) ~[?:1.8.0_201] at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_201] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_201] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_201] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_201] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_201] at java.util.stream.LongPipeline.reduce(LongPipeline.java:438) ~[?:1.8.0_201] at java.util.stream.LongPipeline.sum(LongPipeline.java:396) ~[?:1.8.0_201] at java.util.stream.ReferencePipeline.count(ReferencePipeline.java:526) ~[?:1.8.0_201] at org.graylog2.database.PaginatedDbService.findPaginatedWithQueryFilterAndSort(PaginatedDbService.java:159) ~[graylog.jar:?] at org.graylog2.streams.PaginatedStreamService.findPaginated(PaginatedStreamService.java:48) ~[graylog.jar:?] at org.graylog2.rest.resources.streams.StreamResource.getPage(StreamResource.java:210) ~[graylog.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_201] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_201] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[graylog.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[graylog.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[graylog.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[graylog.jar:?] at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) [graylog.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?] at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) [graylog.jar:?] at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) [graylog.jar:?] at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?] at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?] at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_201] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_201] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201] Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `org.graylog2.streams.StreamDTO`, problem: Null matchingType at [Source: de.undercouch.bson4jackson.io.LittleEndianInputStream@67bcd2e9; pos: 536] at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67) ~[graylog.jar:?] at com.fasterxml.jackson.databind.DeserializationContext.instantiationException(DeserializationContext.java:1608) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapAsJsonMappingException(StdValueInstantiator.java:484) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.rewrapCtorProblem(StdValueInstantiator.java:503) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:285) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith(ValueInstantiator.java:229) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:195) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:488) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1287) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159) ~[graylog.jar:?] at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3985) ~[graylog.jar:?] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2343) ~[graylog.jar:?] at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:80) ~[graylog.jar:?] at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:64) ~[graylog.jar:?] ... 74 more Caused by: java.lang.NullPointerException: Null matchingType at org.graylog2.streams.$AutoValue_StreamDTO.<init>($AutoValue_StreamDTO.java:71) ~[graylog.jar:?] at org.graylog2.streams.AutoValue_StreamDTO.<init>(AutoValue_StreamDTO.java:19) ~[graylog.jar:?] at org.graylog2.streams.StreamDTO.create(StreamDTO.java:141) ~[graylog.jar:?] at sun.reflect.GeneratedMethodAccessor398.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_201] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_201] at com.fasterxml.jackson.databind.introspect.AnnotatedMethod.call(AnnotatedMethod.java:109) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:283) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.ValueInstantiator.createFromObjectWith(ValueInstantiator.java:229) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:195) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:488) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1287) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326) ~[graylog.jar:?] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159) ~[graylog.jar:?] at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3985) ~[graylog.jar:?] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2343) ~[graylog.jar:?] at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:80) ~[graylog.jar:?] at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:64) ~[graylog.jar:?] ... 74 more
I hope this additional information helps to figure out what the problem is.