Failed to acquire lock on file .lock --Journal

Graylog Central (peer support)
1

Hi All,

Getting this error suddenly without any changes done on graylog server.

2020-03-11T12:30:09.698+05:30 ERROR [KafkaJournal] Unable to start logmanager.
kafka.common.KafkaException: Failed to acquire lock on file .lock in /var/lib/graylog-server/journal. A Kafka instance in another process or thread is using this directory.
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:98) ~[graylog.jar:?]
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95) ~[graylog.jar:?]
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[graylog.jar:?]
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[graylog.jar:?]
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) ~[graylog.jar:?]
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) ~[graylog.jar:?]
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) ~[graylog.jar:?]
at scala.collection.AbstractTraversable.map(Traversable.scala:104) ~[graylog.jar:?]
at kafka.log.LogManager.lockLogDirs(LogManager.scala:95) ~[graylog.jar:?]
at kafka.log.LogManager.(LogManager.scala:57) ~[graylog.jar:?]
at org.graylog2.shared.journal.KafkaJournal.(KafkaJournal.java:326) [graylog.jar:?]
at org.graylog2.shared.journal.KafkaJournal.(KafkaJournal.java:191) [graylog.jar:?]
at org.graylog2.shared.journal.KafkaJournal$$FastClassByGuice$$8f50433e.newInstance() [graylog.jar:?]
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:148) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:148) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:148) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:148) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) [graylog.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) [graylog.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) [graylog.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) [graylog.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [graylog.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:148) [graylog.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39) [graylog.jar:?]
at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:211) [graylog.jar:?]
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:182) [graylog.jar:?]
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109) [graylog.jar:?]
at com.google.inject.Guice.createInjector(Guice.java:87) [graylog.jar:?]
at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:379) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:194) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:50) [graylog.jar:?]

I have tried changing the journal data path but still same issue. Graylog works fine after deleting the journal folder but again after sometime I am getting the above logs.

I have confirmed that no other java process is running apart from Graylog. Kindly help. My graylog is not working for more than 30 mins.

1 Like
2

Adding logs I get in server.log.

2020-03-11T18:19:49.660+05:30 ERROR [LogManager] There was an error in one of the threads during logs loading: java.lang.IllegalArgumentException: requirement failed: Attempt to append to a full index (size = 131072).

2020-03-11T18:19:49.732+05:30 INFO [InputBufferImpl] Message journal is enabled.

2020-03-11T18:19:49.737+05:30 ERROR [KafkaJournal] Unable to start logmanager.
kafka.common.KafkaException: Failed to acquire lock on file .lock in /graylog/journal_data/journal. A Kafka instance in another process or thread is using this directory.
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:98) ~[graylog.jar:?]
at kafka.log.LogManager$$anonfun$lockLogDirs$1.apply(LogManager.scala:95) ~[graylog.jar:?]
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[graylog.jar:?]
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[graylog.jar:?]
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) ~[graylog.jar:?]
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) ~[graylog.jar:?]
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) ~[graylog.jar:?]

Kindly help . I am stuck.

3

@macko003 @jan Can you please help. Thanks in advance.

4

Hi!

Which value did you set for message_journal_max_size in server.conf?

Seems to be a disk space issue, check if there is space available on the machine running the app.

5

There is enough disk space. I have verified the same.
message_journal_max_size is set to message_journal_max_size = 50gb.

/devmessage_journal_dir = /graylog/journal_data/journal

Disk space:

/sdb1 22T 1.2T 20T 6% /graylog

I have even changed the directory to /graylog which is 20TB free.

6

  1. It’s a community support. And I don’t get salary for solve other’s problem.

  2. I see only, what you see also. Read your log.
    (the quoted line is the 2nd line)
    Graylog can’t access and lock the /var/lib/graylog-server/journal folder.
    Solve this problem.

Are you sure it is in your server.conf?

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.