Graylog service active but application not running

Hey! I am looking for help bringing Graylog back to life. It somehow crashed due to a power outage. Since then it does not start correctly. I am probably overlooking something. I suspect an error with jvm settings.

I upgraded Graylog to 4.3 already (on Ubuntu 18.04).
Elasticsearch version is 7.10.2.

Service status

admin@ubuntu_1804:~$ sudo systemctl status graylog-server.service
● graylog-server.service - Graylog server
   Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2022-11-18 15:16:05 CET; 28s ago
     Docs: http://docs.graylog.org/
 Main PID: 7629 (graylog-server)
    Tasks: 37 (limit: 4915)
   CGroup: /system.slice/graylog-server.service
           ├─7629 /bin/sh /usr/share/graylog-server/bin/graylog-server
           └─7676 /usr/bin/java -Xms2g -Xmx2g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSC

Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]: End of classname legend:
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]: ========================
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:568)
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:190)
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:113)
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at com.google.inject.Guice.createInjector(Guice.java:87)
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34)
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:490)
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:295)
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at org.graylog2.bootstrap.Main.main(Main.java:45)

Graylog server log

2022-11-18T15:16:08.808+01:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2022-11-18T15:16:10.985+01:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 4.3.9 [org.graylog.aws.AWSPlugin]
2022-11-18T15:16:10.988+01:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integrations 4.3.9 [org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin]
2022-11-18T15:16:10.990+01:00 INFO  [CmdLineTool] Loaded plugin: Integrations 4.3.9 [org.graylog.integrations.IntegrationsPlugin]
2022-11-18T15:16:10.992+01:00 INFO  [CmdLineTool] Loaded plugin: Collector 4.3.9 [org.graylog.plugins.collector.CollectorPlugin]
2022-11-18T15:16:10.995+01:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise 4.3.9 [org.graylog.plugins.enterprise.EnterprisePlugin]
2022-11-18T15:16:10.998+01:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES6 Support) 4.3.9 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es6.EnterpriseES6Plugin]
2022-11-18T15:16:11.000+01:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES7 Support) 4.3.9 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es7.EnterpriseES7Plugin]
2022-11-18T15:16:11.003+01:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.3.9 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2022-11-18T15:16:11.004+01:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.3.9+e2c6648 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2022-11-18T15:16:11.005+01:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.3.9+e2c6648 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2022-11-18T15:16:11.109+01:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms2g -Xmx2g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2022-11-18T15:16:13.075+01:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2022-11-18T15:16:13.300+01:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-11-18T15:16:13.470+01:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:34}] to localhost:27017
2022-11-18T15:16:13.489+01:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 28]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=9325632}
2022-11-18T15:16:13.540+01:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:35}] to localhost:27017
2022-11-18T15:16:13.628+01:00 INFO  [connection] Closed connection [connectionId{localValue:2, serverValue:35}] to localhost:27017 because the pool has been closed.
2022-11-18T15:16:13.637+01:00 INFO  [MongoDBPreflightCheck] Connected to MongoDB version 4.0.28
2022-11-18T15:16:13.967+01:00 INFO  [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <Elasticsearch:7.10.2>
2022-11-18T15:16:14.291+01:00 INFO  [Version] HV000001: Hibernate Validator null
2022-11-18T15:16:25.596+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:25.651+01:00 INFO  [NodeId] Node ID: c251c81e-9729-4ec5-9348-7c7132fe8d8d
2022-11-18T15:16:26.202+01:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2022-11-18T15:16:26.206+01:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-11-18T15:16:26.214+01:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:36}] to localhost:27017
2022-11-18T15:16:26.217+01:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 28]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1509000}
2022-11-18T15:16:26.224+01:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:37}] to localhost:27017
2022-11-18T15:16:27.241+01:00 INFO  [ElasticsearchVersionProvider] Elasticsearch cluster is running Elasticsearch:7.10.2
2022-11-18T15:16:28.685+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:29.185+01:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:38}] to localhost:27017
2022-11-18T15:16:29.278+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:30.034+01:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-11-18T15:16:30.068+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:30.104+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:31.129+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:31.687+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:31.694+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:31.706+01:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-11-18T15:16:31.715+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:31.803+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:31.851+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:31.883+01:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): java.lang.NumberFormatException: For input string: ""
2022-11-18T15:16:31.883+01:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): java.lang.NumberFormatException: For input string: ""
2022-11-18T15:16:31.884+01:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): java.lang.NumberFormatException: For input string: ""
2022-11-18T15:16:31.884+01:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): java.lang.NumberFormatException: For input string: ""
2022-11-18T15:16:31.884+01:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): java.lang.NumberFormatException: For input string: ""

/etc/default/graylog-server

# Path to the java executable.
JAVA=/usr/bin/java

# Default Java options for heap and garbage collection.
GRAYLOG_SERVER_JAVA_OPTS="-Xms2g -Xmx2g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow"

# Avoid endless loop with some TLSv1.3 implementations.
GRAYLOG_SERVER_JAVA_OPTS="$GRAYLOG_SERVER_JAVA_OPTS -Djdk.tls.acknowledgeCloseNotify=true"

# Fix for log4j CVE-2021-44228
GRAYLOG_SERVER_JAVA_OPTS="$GRAYLOG_SERVER_JAVA_OPTS -Dlog4j2.formatMsgNoLookups=true"

# Pass some extra args to graylog-server. (i.e. "-d" to enable debug mode)
#GRAYLOG_SERVER_ARGS=""

# Program that will be used to wrap the graylog-server command. Useful to
# support programs like authbind.
#GRAYLOG_COMMAND_WRAPPER=""

Thanks for you help!

Hello @dani

I’m assuming this is just one node with ES, GL & MongoDb on it? By chance do you have another volume attached to the node?

This line here.

 ERROR [CmdLineTool] Guice error (more detail on log level debug): java.lang.NumberFormatException: For input string: ""

Edit: Is that the only error you have in Graylog log file?
Since you stated JAVA what version are you using?
What trouble shooting have you done?
Sometimes it could be that the journal got corrupted, But I’m not seeing that error it in the log files. It shows “input string”.

EDIT:2
Doing some research on this, I’m wonder if it may have something to do with your inputs/Indices. Are you able to manually rotate your indices and restart Graylog service? Don’t for get to tail log file.

By the way pretty much everything runs on java in Graylog :laughing:

The reason I stated this was from here “java.lang.NumberFormatException:”.

https://docs.oracle.com/javase/8/docs/api/java/lang/NumberFormatException.html

TBH since this server crashed, if you don’t mind losing some data/messages. Stop Graylog service clean out the journal then restart Graylog service again. I had to do this a couple time in the past. Tail the log file see if that clears things up.

2 Likes

Hello @gsmith

Yes, I have just one node with ES, GL & Mongo DB and no other volume.
Can’t find other errors. With JAVA I actually meant JVM arguments in this line:

2022-11-18T15:16:11.109+01:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms2g -Xmx2g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb

Troubleshooting so far:

  • checking my configuration files
  • reverting changes for java heap sizes, as I had troubles with that previously
  • purged elasticsearch and made a new install (I upgraded once to elasticsearch 7.11 and it worked for a while - know I shouldn’t do that)
  • upgraded Graylog to 4.3

But your hint “Stop Graylog service clean out the journal then restart Graylog service again.” did it!!
Now I have some Elasticsearch errors to investigate, but I can log on to Graylog again.

Thanks for your help!

1 Like

Hey @dani

awesome-yes-will-ferrell (1)

Glad I could help, If you could mark your solution resolved for future searches that would be great :+1: