Graylog-server service won't start

Graylog Central (peer support)
1

In a test environment, the graylog server service won’t start for some reason which I assume is from the journal size
I’ve already deleted the journal with no luck
Reinstall graylog 4.2 and 4.3 with the same result on
Ubuntu 20.04 on a VM with 8GB RAM and 60GB of HDD

Service logs, configurations, and environment variables

is_leader = true
node_id_file = /etc/graylog/server/node-id
password_secret = QmzmOUCm9UZTKcesJfuOAi6K9ut8T2Ebh53o4IwnuOs628Jyn3lu52E3fisFEc35HWXGLr5CxeDlOCEcZR8CY91eqj1eFGHf
root_password_sha2 = 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 192.168.1.47:9000
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

status:

 graylog-server.service - Graylog server
     Loaded: loaded (/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Wed 2022-06-01 10:44:13 UTC; 5s ago
       Docs: http://docs.graylog.org/
    Process: 9913 ExecStart=/usr/share/graylog-server/bin/graylog-server (code=exited, status=1/FAILURE)
   Main PID: 9913 (code=exited, status=1/FAILURE)

Jun 01 10:44:13 graylog systemd[1]: graylog-server.service: Main process exited, code=exited, status=1/FAILURE
Jun 01 10:44:13 graylog systemd[1]: graylog-server.service: Failed with result 'exit-code'.

and the log:

2022-06-01T10:45:49.963Z INFO  [CmdLineTool] Running with JVM arguments: -Xms2g -Xmx2g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2022-06-01T10:45:50.304Z INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2022-06-01T10:45:50.330Z INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-06-01T10:45:50.352Z INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:243}] to localhost:27017
2022-06-01T10:45:50.358Z INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 28]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=4532996}
2022-06-01T10:45:50.367Z INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:244}] to localhost:27017
2022-06-01T10:45:50.383Z INFO  [connection] Closed connection [connectionId{localValue:2, serverValue:244}] to localhost:27017 because the pool has been closed.
2022-06-01T10:45:50.385Z INFO  [MongoDBPreflightCheck] Connected to MongoDB version 4.0.28
2022-06-01T10:45:50.461Z INFO  [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <Elasticsearch:7.10.2>
2022-06-01T10:45:50.480Z ERROR [PreflightCheckService] Preflight check failed with error: Journal directory </var/lib/graylog-server/journal> has not enough free space (2614 MB) to contain 'message_journal_max_size = 5120 MB'

Any input, help , suggestions are highly appreciated

Thank you
Andreas

2

Hello && Wlecome @andresorensen

Looks like you have a default setup.

I did adjust this post so its easier to read. If you could use the </> markdown that would be great. It makes Logs and configuration files easier to read , hence help you.

Have you tried increasing the journal size? ( This is done in Graylog configuration file)

#message_journal_max_age = 12h
message_journal_max_size = 12gb

#message_journal_flush_age = 1m
#message_journal_flush_interval = 1000000
#message_journal_segment_age = 1h

Enough disk space on this node?

3

@gsmith Thank you very much for the input
After modifying the journal size accordingly, the server started without any more issues
Really appreciate your help

1 Like
4

@andresorensen
Nice glad I could help.
If you could mark this as resolved that would be great :+1:

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.