How to move graylog_journal to new location? (Docker environment)

[schneich (Chris)]

1. Describe your incident:
Dear community,

I am playing around with a graylog container. In my first steps, Graylog was running okay, but I got the paths in my compose.yaml wrong.

I stoped all containers and then copied the files from the source location to then new target location.

sudo cp -arT /var/lib/docker/volumes/graylog_es_data /home/uadmin/Docker/Graylog/es_data
sudo cp -arT /var/lib/docker/volumes/graylog_graylog_journal /home/uadmin/Docker/Graylog/graylog_journal
sudo cp -arT /var/lib/docker/volumes/graylog_mongo_data /home/uadmin/Docker/Graylog/mongo_data

All data has been successfully copied.

Then, I changed my compose.yaml to this:

...
    volumes:
     -  /home/uadmin/Docker/Graylog/graylog_journal:/usr/share/graylog/data/journal
...

When I now start Graylog with the compose.yaml, it is not booting.

The logs for the graylog-container show something about “log manager” and “kafka”. The log has 17’000+ lines, but I thing it is allways the same error. (see bellow).

Warning can not change owner to graylog:graylog
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
...
2022-12-30 23:33:26,331 ERROR: org.graylog.shaded.kafka09.log.LogManager - There was an error in one of the threads during logs loading: org.graylog.shaded.kafka09.common.KafkaException: Found directory /usr/share/graylog/data/journal/_data, '_data' is not in the form of topic-partition
If a directory does not contain Kafka topic data it should not exist in Kafka's log directory
2022-12-30 23:33:26,332 ERROR: org.graylog2.shared.journal.LocalKafkaJournal - Unable to start logmanager.
org.graylog.shaded.kafka09.common.KafkaException: Found directory /usr/share/graylog/data/journal/_data, '_data' is not in the form of topic-partition
If a directory does not contain Kafka topic data it should not exist in Kafka's log directory
	at org.graylog.shaded.kafka09.log.Log$.throwException(Log.scala:948) ~[graylog.jar:?]
	at org.graylog.shaded.kafka09.log.Log$.parseTopicPartitionName(Log.scala:936) ~[graylog.jar:?]
	at org.graylog.shaded.kafka09.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:146) ~[graylog.jar:?]
	at org.graylog.shaded.kafka09.utils.CoreUtils$$anon$1.run(CoreUtils.scala:60) ~[graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
...
2022-12-30 23:33:26,362 INFO : org.graylog.shaded.kafka09.log.LogManager - Loading logs.
2022-12-30 23:33:26,363 ERROR: org.graylog.shaded.kafka09.log.LogManager - There was an error in one of the threads during logs loading: org.graylog.shaded.kafka09.common.KafkaException: Found directory /usr/share/graylog/data/journal/_data, '_data' is not in the form of topic-partition
If a directory does not contain Kafka topic data it should not exist in Kafka's log directory
2022-12-30 23:33:26,363 ERROR: org.graylog2.shared.journal.LocalKafkaJournal - Unable to start logmanager.
org.graylog.shaded.kafka09.common.KafkaException: Found directory /usr/share/graylog/data/journal/_data, '_data' is not in the form of topic-partition
If a directory does not contain Kafka topic data it should not exist in Kafka's log directory
	at org.graylog.shaded.kafka09.log.Log$.throwException(Log.scala:948) ~[graylog.jar:?]
	at org.graylog.shaded.kafka09.log.Log$.parseTopicPartitionName(Log.scala:936) ~[graylog.jar:?]
	at org.graylog.shaded.kafka09.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:146) ~[graylog.jar:?]
	at org.graylog.shaded.kafka09.utils.CoreUtils$$anon$1.run(CoreUtils.scala:60) ~[graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
2022-12-30 23:33:26,367 ERROR: org.graylog2.shared.journal.LocalKafkaJournal - Unable to start logmanager.
org.graylog.shaded.kafka09.common.KafkaException: Failed to acquire lock on file .lock in /usr/share/graylog/data/journal. A Kafka instance in another process or thread is using this directory.
...

2. Describe your environment:

• Ubuntu 22.04.1 LTS
• Docker Engine 20.10.22

3. What steps have you already taken to try and solve the problem?

• There is a .lock-file, which I deleted - to no effekt.
• I tried to provide access rights to all users on the directory “sudo chmod a+rwx /home/uadmin/Docker/Graylog/graylog_journal”

4. How can the community help?
Two questions:

• How do I solve this?
• Is there an official pest practice on how to backup and restore this? I was assuming, I could make a backup of those folders and the compose.yaml and I could then easyly restore everthing.

[schneich (Chris)]

This discussion implies that you cannot rename the folder?

Ran out of disk, now "Failed to acquire lock on file .lock"

I have confirmed only one instance of Graylog is running. And using lsof I have confirmed that no other process is using the .lock file. The journal directory is not on its own disk partition. But that reminded me of something I read in my prior research: someone mentioned that having any extra files in the journal folder would cause this error. I didn’t have any extra files, but I did rename the journal folder as a test. Graylog starts up correctly now. So now the question becomes: how can I…

I did rename the folder, which persists data of graylog…

Well, renaming the folder back did not help. I get the feeling, this might have something todo wich access reights. I read a lot about graylog user 1100. This user does not exist in my host. Do I have to create it manually and provide access to the specific folders? What ID, Name and Group(s) do I have to set up? How do I do that on a Ubuntu Terminal?

[schneich (Chris)]

I created a user graylog with id 1100 and I used the following comand on the directory.

chown -R 1100:1100 ./graylog_journal

(Source: ERROR: Unable to access file /usr/share/graylog/data/journal/graylog2-committed-read-offset: Permission denied · Issue #2155 · Graylog2/graylog2-server · GitHub)

Graylog still not starting, but the logfile only contains 3500+ lines - a missive impruvement.
This error message repeats over and over again:

...
2022-12-31 22:07:00,271 ERROR: org.graylog.shaded.kafka09.log.LogManager - There was an error in one of the threads during logs loading: org.graylog.shaded.kafka09.common.KafkaException: Found directory /usr/share/graylog/data/journal/_data, '_data' is not in the form of topic-partition
If a directory does not contain Kafka topic data it should not exist in Kafka's log directory
2022-12-31 22:07:00,271 ERROR: org.graylog2.shared.journal.LocalKafkaJournal - Unable to start logmanager.
org.graylog.shaded.kafka09.common.KafkaException: Found directory /usr/share/graylog/data/journal/_data, '_data' is not in the form of topic-partition
If a directory does not contain Kafka topic data it should not exist in Kafka's log directory
	at org.graylog.shaded.kafka09.log.Log$.throwException(Log.scala:948) ~[graylog.jar:?]
	at org.graylog.shaded.kafka09.log.Log$.parseTopicPartitionName(Log.scala:936) ~[graylog.jar:?]
	at org.graylog.shaded.kafka09.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:146) ~[graylog.jar:?]
	at org.graylog.shaded.kafka09.utils.CoreUtils$$anon$1.run(CoreUtils.scala:60) ~[graylog.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
...

[schneich (Chris)]

I figured this out myself:

the original path, which was created Docker, was this:

/var/lib/docker/volumes/graylog_es_data/_data
/var/lib/docker/volumes/graylog_graylog_journal/_data
/var/lib/docker/volumes/graylog_mongo_data/_data

Because I copied the “_data”-folder, there were the above error messages. After removing the “_data”-folders and putting everything into the correct paths, Graylog is running as it should be.