I have deployed a new two node cluster with Graylog 3.2.5. Everything is working fine, but calling the /system/cluster/traffic throws an error on both nodes.
Response:
“type”: “ApiError”,
“message”: “IOException encountered while reading from a byte array input stream”
There might be an error calculating outgoing traffic, as this information does not show up on weg page System -> Overview. It stay on “loading”. I dont have configured any Outputs.
sorry, it wasnt in comp. mode. I was using “latest” docs which say Graylog is compatible with mongodb 4.2. But latest means Graylog 3.3 and not my latest version 3.2.5.
I set compat. mode to 4.0 in mongodb. Then I restartet MongoDB and Graylog Server but the issue is still there. Do I have to wait or do anything different? Compat mode for 4.0 is active.
I updated graylog to 3.3.0 - same issue.
Then I disabled MongoDB compat. mode, so MongoDB 4.2 is running at 4.2 feature level - same issue.
But i found some errors in the logs. They do repeat every minute.
MongoDB
2020-05-26T15:17:33.016+0200 I ACCESS [conn10] Unauthorized: not authorized on admin to execute command { serverStatus: true, $db: "admin", $clusterTime: { clusterTime: Timestamp(1590499052, 6), signature: { hash: BinData(0, 393F9D7583413743FB568EAC9AFF01027D2FF36C), keyId: 6828817852531736579 } }, lsid: { id: UUID("0806a547-2e1c-414d-9c4b-073d5bfb2051") } }
Graylog
2020-05-26T15:17:26.660+02:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:67) ~[graylog.jar:?]
at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:49) ~[graylog.jar:?]
at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:29) ~[graylog.jar:?]
at com.mongodb.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52) ~[graylog.jar:?]
at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:60) ~[graylog.jar:?]
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84) ~[graylog.jar:?]
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41) ~[graylog.jar:?]
at org.bson.internal.LazyCodec.decode(LazyCodec.java:48) ~[graylog.jar:?]
at org.bson.codecs.BsonDocumentCodec.readValue(BsonDocumentCodec.java:101) ~[graylog.jar:?]
at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:63) ~[graylog.jar:?]
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84) ~[graylog.jar:?]
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41) ~[graylog.jar:?]
at com.mongodb.internal.connection.ReplyMessage.<init>(ReplyMessage.java:51) ~[graylog.jar:?]
at com.mongodb.internal.connection.InternalStreamConnection.getCommandResult(InternalStreamConnection.java:413) ~[graylog.jar:?]
at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:309) ~[graylog.jar:?]
at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:259) ~[graylog.jar:?]
at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:450) ~[graylog.jar:?]
at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:226) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131) ~[graylog.jar:?]
at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123) ~[graylog.jar:?]
at com.mongodb.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:343) ~[graylog.jar:?]
at com.mongodb.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:334) ~[graylog.jar:?]
at com.mongodb.operation.CommandOperationHelper.executeCommandWithConnection(CommandOperationHelper.java:220) ~[graylog.jar:?]
at com.mongodb.operation.FindOperation$1.call(FindOperation.java:731) ~[graylog.jar:?]
at com.mongodb.operation.FindOperation$1.call(FindOperation.java:725) ~[graylog.jar:?]
at com.mongodb.operation.OperationHelper.withReadConnectionSource(OperationHelper.java:463) ~[graylog.jar:?]
at com.mongodb.operation.FindOperation.execute(FindOperation.java:725) ~[graylog.jar:?]
at com.mongodb.operation.FindOperation.execute(FindOperation.java:89) ~[graylog.jar:?]
at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:196) ~[graylog.jar:?]
at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:177) ~[graylog.jar:?]
at com.mongodb.DBCursor.initializeCursor(DBCursor.java:989) ~[graylog.jar:?]
at com.mongodb.DBCursor.hasNext(DBCursor.java:172) ~[graylog.jar:?]
at org.mongojack.DBCursor.hasNext(DBCursor.java:330) ~[graylog.jar:?]
at java.lang.Iterable.forEach(Iterable.java:74) ~[?:1.8.0_252]
at org.graylog2.system.traffic.TrafficCounterService.clusterTrafficOfLastDays(TrafficCounterService.java:102) ~[graylog.jar:?]
at org.graylog2.rest.resources.system.TrafficResource.get(TrafficResource.java:56) ~[graylog.jar:?]
at sun.reflect.GeneratedMethodAccessor398.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.lang.Long` out of START_OBJECT token
at [Source: de.undercouch.bson4jackson.io.LittleEndianInputStream@1892303b; pos: 149] (through reference chain: org.graylog2.system.traffic.AutoValue_TrafficDto$Builder["decoded"]->java.util.LinkedHashMap["myhostname"])
at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1343) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1139) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1093) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.std.NumberDeserializers$LongDeserializer._parseLong(NumberDeserializers.java:593) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.std.NumberDeserializers$LongDeserializer.deserialize(NumberDeserializers.java:557) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.std.NumberDeserializers$LongDeserializer.deserialize(NumberDeserializers.java:535) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:527) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:364) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeSetAndReturn(MethodProperty.java:158) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.vanillaDeserialize(BuilderBasedDeserializer.java:269) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:193) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3985) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2343) ~[graylog.jar:?]
at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:80) ~[graylog.jar:?]
at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:64) ~[graylog.jar:?]
... 64 more
So it seems that Graylog try to connect to mongodb without authentication. But it does not show which command it tries to execute.
I have created a graylog db in MongoDB with user graylog and roles dbAdmin + readWrite.
I did a test with mongo -u graylog -p xxx on every node, this was ok.
Today I removed all data (wiped /var/lib/elasticsearch, /var/lib/mongo, /var/lib/graylog-server) from the system, removed config/apps and did a clean install. After that now I am facing the same issue again.
Then I disabled mongo replset and auth, to use a singe mongodb node. I dont have errors in mongodb log. So I think the authentication is working. But the issue isnt gone.
I do not very much about Java, but I had a look at the source file causing this NPE.
In line 212 of file src/main/java/org/graylog2/system/stats/mongo/MongoProbe.java gl tries to collect memory stats. I dont know java, so I cant understand whats going wrong here…
Then I found a PR on github changing this file a few years ago to fix a NPE. This fix made stat mem. mappedWithJournal optional, as this is only in MongoDB serverStatus output if you use MMAPv1 storage engine. But based on the MongoDB docs, the stat mem.mapped (which is causing the NPE in my case) its the same. So I think stat mem.mapped also needs to be optional in serverStatus output. I created a issue on Github.
