Can you share more details? What version of graylog are you running? Is this a new cluster that has never worked and you are getting this error for the first time or is this an existing cluster that used to work and now you receive this error?
Feb 08 08:55:52 graylog graylog-server[18990]: at org.graylog2.bootstrap.Main.main(Main.java:50)
Feb 08 08:55:52 graylog graylog-server[18990]: 10 errors
Feb 08 08:55:52 graylog graylog-server[18990]: at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543)
Feb 08 08:55:52 graylog graylog-server[18990]: at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
Feb 08 08:55:52 graylog graylog-server[18990]: at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
Feb 08 08:55:52 graylog graylog-server[18990]: at com.google.inject.Guice.createInjector(Guice.java:87)
Feb 08 08:55:52 graylog graylog-server[18990]: at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34)
Feb 08 08:55:52 graylog graylog-server[18990]: at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:379)
Feb 08 08:55:52 graylog graylog-server[18990]: at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:194)
Feb 08 08:55:52 graylog graylog-server[18990]: at org.graylog2.bootstrap.Main.main(Main.java:50)
Some data form server.log
2024-02-07T16:41:25.984+08:00 ERROR [KafkaJournal] Unable to start logmanager.
kafka.common.KafkaException: Failed to acquire lock on file .lock in /var/lib/graylog-server/journal. A Kafka instance in another process or thread is using this directory.
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:98) ~[graylog.jar:?]
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95) ~[graylog.jar:?]
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[graylog.jar:?]
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[graylog.jar:?]
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) ~[graylog.jar:?]
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) ~[graylog.jar:?]
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) ~[graylog.jar:?]
at scala.collection.AbstractTraversable.map(Traversable.scala:104) ~[graylog.jar:?]
at kafka.log.LogManager.lockLogDirs(LogManager.scala:95) ~[graylog.jar:?]
at kafka.log.LogManager.(LogManager.scala:57) ~[graylog.jar:?]
at org.graylog2.shared.journal.KafkaJournal.(KafkaJournal.java:326) [graylog.jar:?]
at org.graylog2.shared.journal.KafkaJournal.(KafkaJournal.java:191) [graylog.jar:?]
at org.graylog2.shared.journal.KafkaJournal$$FastClassByGuice$$8f50433e.newInstance() [graylog.jar:?]
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:211) [graylog.jar:?]
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:182) [graylog.jar:?]
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109) [graylog.jar:?]
at com.google.inject.Guice.createInjector(Guice.java:87) [graylog.jar:?]
at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:379) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:194) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:50) [graylog.jar:?]
2024-02-07T16:41:26.012+08:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2024-02-07T16:41:26.013+08:00 ERROR [KafkaJournal] Unable to start logmanager.
kafka.common.KafkaException: Failed to acquire lock on file .lock in /var/lib/graylog-server/journal. A Kafka instance in another process or thread is using this directory.
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:98) ~[graylog.jar:?]
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95) ~[graylog.jar:?]
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[graylog.jar:?]
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[graylog.jar:?]
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) ~[graylog.jar:?]
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) ~[graylog.jar:?]
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) ~[graylog.jar:?]
at scala.collection.AbstractTraversable.map(Traversable.scala:104) ~[graylog.jar:?]
at kafka.log.LogManager.lockLogDirs(LogManager.scala:95) ~[graylog.jar:?]
at kafka.log.LogManager.(LogManager.scala:57) ~[graylog.jar:?]
at org.graylog2.shared.journal.KafkaJournal.(KafkaJournal.java:326) [graylog.jar:?]
at org.graylog2.shared.journal.KafkaJournal.(KafkaJournal.java:191) [graylog.jar:?]
at org.graylog2.shared.journal.KafkaJournal$$FastClassByGuice$$8f50433e.newInstance() [graylog.jar:?]
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050) [graylog.jar:?]
at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:836) [graylog.jar:?]
at com.sun.proxy.$Proxy31.create(Unknown Source) [?:?]
at org.graylog2.shared.buffers.ProcessBuffer.(ProcessBuffer.java:91) [graylog.jar:?]
at org.graylog2.shared.buffers.ProcessBuffer$$FastClassByGuice$$ef94431e.newInstance() [graylog.jar:?]
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
Graylog Version : 4.2.3
Elasticsearch : 7.17.9
MongoDB : 4.4.8
LOGS :
2024-02-15T12:42:31.162+05:30 INFO [CmdLineTool] Loaded plugin: AWS plugins 4.0.17 [org.graylog.aws.AWSPlugin]
2024-02-15T12:42:31.174+05:30 INFO [CmdLineTool] Loaded plugin: Collector 4.0.17 [org.graylog.plugins.collector.CollectorPlugin]
2024-02-15T12:42:31.176+05:30 INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.0.17 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2024-02-15T12:42:31.176+05:30 INFO [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.0.17+d0c5b22 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2024-02-15T12:42:31.176+05:30 INFO [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.0.17+d0c5b22 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2024-02-15T12:42:31.907+05:30 INFO [CmdLineTool] Running with JVM arguments: -Xms12g -Xmx12g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2024-02-15T12:42:32.615+05:30 INFO [Version] HV000001: Hibernate Validator null
2024-02-15T12:42:38.992+05:30 INFO [InputBufferImpl] Message journal is enabled.
2024-02-15T12:42:39.053+05:30 INFO [NodeId] Node ID: 5ba59c58-f3fa-4e04-b3b4-15b5d6c311c2
2024-02-15T12:42:39.717+05:30 INFO [LogManager] Loading logs.
2024-02-15T12:42:39.770+05:30 WARN [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000000000000000.index, deleting and rebuilding index…
2024-02-15T12:42:39.847+05:30 INFO [LogManager] Logs loading complete.
2024-02-15T12:42:39.853+05:30 INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2024-02-15T12:42:39.913+05:30 INFO [cluster] Cluster created with settings {hosts=[192.168.24.28:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2024-02-15T12:42:39.993+05:30 INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2024-02-15T12:42:40.038+05:30 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:262}] to 192.168.24.28:27017
2024-02-15T12:42:40.047+05:30 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=192.168.24.28:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 8]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=6215893}
2024-02-15T12:42:40.070+05:30 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:263}] to 192.168.24.28:27017
2024-02-15T12:42:40.419+05:30 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.
2024-02-15T12:42:41.263+05:30 INFO [ElasticsearchVersionProvider] Elasticsearch cluster is running v7.17.9
2024-02-15T12:42:42.622+05:30 INFO [connection] Opened connection [connectionId{localValue:3, serverValue:264}] to 192.168.24.28:27017
2024-02-15T12:42:42.636+05:30 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2024-02-15T12:42:43.025+05:30 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2024-02-15T12:42:43.232+05:30 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2024-02-15T12:42:43.452+05:30 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2024-02-15T12:42:43.631+05:30 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2024-02-15T12:42:44.175+05:30 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2024-02-15T12:42:44.343+05:30 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2024-02-15T12:42:44.687+05:30 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2024-02-15T12:42:44.963+05:30 INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy .
2024-02-15T12:42:44.980+05:30 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2024-02-15T12:42:45.147+05:30 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2024-02-15T12:42:45.296+05:30 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
2024-02-15T12:42:45.297+05:30 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
2024-02-15T12:42:45.297+05:30 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
2024-02-15T12:42:45.297+05:30 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
2024-02-15T12:42:45.297+05:30 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
2024-02-15T12:42:45.297+05:30 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
2024-02-15T12:42:45.297+05:30 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
2024-02-15T12:42:45.298+05:30 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
2024-02-15T12:42:45.298+05:30 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
2024-02-15T12:42:45.298+05:30 ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.RuntimeException: IOException encountered while reading from a byte array input stream
@kedar Your version of Elastic is not supported. Could be due to that.
Also Graylog versions 4.x are out of support. If possible, upgrade to a recent version.
Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000000000000000.index
leads me to believe there are corrupted graylog journal files. I believe we have since implemented better recovery logic but as patrick said above, 4.x is old and out of support so won’t receive those fixes. The workaround is to stop graylog and fully delete the journal directory:
