Graylog Server shutting down on start - Lookup Table


(nqnzp) #1

On start, something appears to fail with loading a lookup table. This seems to be the relevant part of the server.log.

2018-04-05T12:31:33.597-04:00 ERROR [ServiceManager] Service LookupTableService [FAILED] has failed in the STARTING state.
java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
	at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:67) ~[graylog.jar:?]
	at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:49) ~[graylog.jar:?]
	at com.mongodb.DBDecoderAdapter.decode(DBDecoderAdapter.java:29) ~[graylog.jar:?]
	at com.mongodb.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52) ~[graylog.jar:?]
	at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:60) ~[graylog.jar:?]
	at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84) ~[graylog.jar:?]
	at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41) ~[graylog.jar:?]
	at org.bson.codecs.configuration.LazyCodec.decode(LazyCodec.java:47) ~[graylog.jar:?]
	at org.bson.codecs.BsonDocumentCodec.readValue(BsonDocumentCodec.java:101) ~[graylog.jar:?]
	at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:63) ~[graylog.jar:?]
	at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84) ~[graylog.jar:?]
	at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41) ~[graylog.jar:?]
	at com.mongodb.connection.ReplyMessage.<init>(ReplyMessage.java:50) ~[graylog.jar:?]
	at com.mongodb.connection.CommandProtocol.getResponseDocument(CommandProtocol.java:132) ~[graylog.jar:?]
	at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:111) ~[graylog.jar:?]
	at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159) ~[graylog.jar:?]
	at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:289) ~[graylog.jar:?]
	at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:176) ~[graylog.jar:?]
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:216) ~[graylog.jar:?]
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:207) ~[graylog.jar:?]
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:113) ~[graylog.jar:?]
	at com.mongodb.operation.FindOperation$1.call(FindOperation.java:715) ~[graylog.jar:?]
	at com.mongodb.operation.FindOperation$1.call(FindOperation.java:709) ~[graylog.jar:?]
	at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:433) ~[graylog.jar:?]
	at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:406) ~[graylog.jar:?]
	at com.mongodb.operation.FindOperation.execute(FindOperation.java:709) ~[graylog.jar:?]
	at com.mongodb.operation.FindOperation.execute(FindOperation.java:81) ~[graylog.jar:?]
	at com.mongodb.Mongo.execute(Mongo.java:810) ~[graylog.jar:?]
	at com.mongodb.Mongo$2.execute(Mongo.java:797) ~[graylog.jar:?]
	at com.mongodb.DBCursor.initializeCursor(DBCursor.java:871) ~[graylog.jar:?]
	at com.mongodb.DBCursor.hasNext(DBCursor.java:142) ~[graylog.jar:?]
	at org.mongojack.DBCursor.hasNext(DBCursor.java:330) ~[graylog.jar:?]
	at java.util.Iterator.forEachRemaining(Iterator.java:115) ~[?:1.8.0_161]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_161]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_161]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_161]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_161]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_161]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_161]
	at org.graylog2.lookup.LookupTableService.createAndStartAdapters(LookupTableService.java:274) ~[graylog.jar:?]
	at org.graylog2.lookup.LookupTableService.startUp(LookupTableService.java:118) ~[graylog.jar:?]
	at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
	at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of org.graylog.plugins.threatintel.adapters.spamhaus.$AutoValue_SpamhausEDROPDataAdapter_Config$Builder, problem: Missing required properties: refreshInterval
 at [Source: de.undercouch.bson4jackson.io.LittleEndianInputStream@54683a2e; pos: 441] (through reference chain: org.graylog2.lookup.dto.$AutoValue_DataAdapterDto$Builder["config"])
	at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:277) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.DeserializationContext.instantiationException(DeserializationContext.java:1441) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.DeserializationContext.handleInstantiationProblem(DeserializationContext.java:1055) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapInstantiationProblem(BeanDeserializerBase.java:1656) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.finishBuild(BuilderBasedDeserializer.java:137) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:178) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:129) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:209) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:502) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeSetAndReturn(MethodProperty.java:116) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.vanillaDeserialize(BuilderBasedDeserializer.java:266) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:154) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3786) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2181) ~[graylog.jar:?]
	at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:77) ~[graylog.jar:?]
	at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:64) ~[graylog.jar:?]
	... 43 more
Caused by: java.lang.IllegalStateException: Missing required properties: refreshInterval
	at org.graylog.plugins.threatintel.adapters.spamhaus.$AutoValue_SpamhausEDROPDataAdapter_Config$Builder.build($AutoValue_SpamhausEDROPDataAdapter_Config.java:95) ~[?:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
	at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.finishBuild(BuilderBasedDeserializer.java:135) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:178) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:129) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:209) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:502) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeSetAndReturn(MethodProperty.java:116) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.vanillaDeserialize(BuilderBasedDeserializer.java:266) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.deser.BuilderBasedDeserializer.deserialize(BuilderBasedDeserializer.java:154) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3786) ~[graylog.jar:?]
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2181) ~[graylog.jar:?]
	at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:77) ~[graylog.jar:?]
	at org.mongojack.internal.stream.JacksonDBDecoder.decode(JacksonDBDecoder.java:64) ~[graylog.jar:?]
	... 43 more
2018-04-05T12:31:33.608-04:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Failed [LB:DEAD]
2018-04-05T12:31:33.609-04:00 ERROR [InputSetupService] Not starting any inputs because lifecycle is: Failed [LB:DEAD]
2018-04-05T12:31:33.611-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2018-04-05T12:31:33.611-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2018-04-05T12:31:33.611-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2018-04-05T12:31:33.611-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2018-04-05T12:31:33.611-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2018-04-05T12:31:33.612-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2018-04-05T12:31:33.612-04:00 INFO  [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.TrafficCounterCalculator].
2018-04-05T12:31:33.612-04:00 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.TrafficCounterCalculator] complete, took <0ms>.
2018-04-05T12:31:33.614-04:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:113925}] to isim-gl-prd2.admsu.montclair.edu:27017
2018-04-05T12:31:33.615-04:00 INFO  [LogManager] Shutting down.
2018-04-05T12:31:33.657-04:00 INFO  [LogManager] Shutdown complete.
2018-04-05T12:31:33.660-04:00 INFO  [JournalReader] Stopping.
2018-04-05T12:31:33.879-04:00 INFO  [JerseyService] Enabling CORS for HTTP endpoint
2018-04-05T12:31:33.905-04:00 INFO  [Buffers] Waiting until all buffers are empty.
2018-04-05T12:31:33.906-04:00 INFO  [Buffers] All buffers are empty. Continuing.
2018-04-05T12:31:33.907-04:00 INFO  [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2018-04-05T12:31:43.403-04:00 INFO  [NetworkListener] Started listener bound to [isim-gl-prd3.admsu.montclair.edu:12900]
2018-04-05T12:31:43.406-04:00 INFO  [HttpServer] [HttpServer] Started.
2018-04-05T12:31:43.406-04:00 INFO  [JerseyService] Started REST API at <http://isim-gl-prd3.admsu.montclair.edu:12900/>
2018-04-05T12:31:43.408-04:00 INFO  [JerseyService] Enabling CORS for HTTP endpoint
2018-04-05T12:31:44.593-04:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:113926}] to isim-gl-prd2.admsu.montclair.edu:27017
2018-04-05T12:31:44.616-04:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:113927}] to isim-gl-prd2.admsu.montclair.edu:27017
2018-04-05T12:31:45.912-04:00 INFO  [NetworkListener] Started listener bound to [10.0.200.109:9000]
2018-04-05T12:31:45.913-04:00 INFO  [HttpServer] [HttpServer-1] Started.
2018-04-05T12:31:45.913-04:00 INFO  [JerseyService] Started Web Interface at <http://10.0.200.109:9000/>
2018-04-05T12:31:45.917-04:00 INFO  [JerseyService] Shutting down HTTP listener at <http://isim-gl-prd3.admsu.montclair.edu:12900/>
2018-04-05T12:31:45.941-04:00 INFO  [NetworkListener] Stopped listener bound to [isim-gl-prd3.admsu.montclair.edu:12900]
2018-04-05T12:31:45.941-04:00 INFO  [JerseyService] Shutting down HTTP listener at <http://10.0.200.109:9000/>
2018-04-05T12:31:45.947-04:00 INFO  [NetworkListener] Stopped listener bound to [10.0.200.109:9000]
2018-04-05T12:31:45.947-04:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {STARTING=[JerseyService [STARTING], StreamCacheService [STARTING]], FAILED=[LookupTableService [FAILED]]}
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:740) ~[graylog.jar:?]
	at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:553) ~[graylog.jar:?]
	at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:312) ~[graylog.jar:?]
	at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:149) [graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:209) [graylog.jar:?]
	at org.graylog2.bootstrap.Main.main(Main.java:44) [graylog.jar:?]
2018-04-05T12:31:45.948-04:00 INFO  [ServiceManagerListener] Services are now stopped.
2018-04-05T12:31:45.949-04:00 INFO  [Server] SIGNAL received. Shutting down.
2018-04-05T12:31:45.958-04:00 INFO  [GracefulShutdown] Graceful shutdown initiated.
2018-04-05T12:31:45.959-04:00 INFO  [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2018-04-05T12:31:45.959-04:00 WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2018-04-05T12:31:49.970-04:00 INFO  [GracefulShutdown] Goodbye.

(Jochen) #2

Which version of Graylog and which version of the Threat Intelligence plugin are you using?

Have you upgraded Graylog from a beta version or a release candidate?


(nqnzp) #3

Version is most recent stable, 2.4.3, using what I’m assuming is the built-in threat intel plugin. FWIW the other two nodes in my cluster are able to be shut down and restart with no issues.


(nqnzp) #4

Some more information. I discovered that the other two nodes were running version 0.10.0 of the threatintel plugin, and were starting up “correctly” because they were using that plugin, and not the 2.4.3 threatintel plugin that ships with Graylog 2.4.3. Enabling the 2.4.3 threatintel plugin on the other two nodes causes the same error that I posted above.

With threatintel disabled or with the 0.10.0 version deployed I can start up, but obviously I’d like to be on the most recent version. I upgraded from 2.2.1.


(Jochen) #5

You can try deleting the collections related to lookup tables in MongoDB. They all start with lut_.


(system) #6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.