Graylog service active but application not running

dani · November 21, 2022, 5:45pm

Hey! I am looking for help bringing Graylog back to life. It somehow crashed due to a power outage. Since then it does not start correctly. I am probably overlooking something. I suspect an error with jvm settings.

I upgraded Graylog to 4.3 already (on Ubuntu 18.04).
Elasticsearch version is 7.10.2.

Service status

admin@ubuntu_1804:~$ sudo systemctl status graylog-server.service
● graylog-server.service - Graylog server
   Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2022-11-18 15:16:05 CET; 28s ago
     Docs: http://docs.graylog.org/
 Main PID: 7629 (graylog-server)
    Tasks: 37 (limit: 4915)
   CGroup: /system.slice/graylog-server.service
           ├─7629 /bin/sh /usr/share/graylog-server/bin/graylog-server
           └─7676 /usr/bin/java -Xms2g -Xmx2g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSC

Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]: End of classname legend:
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]: ========================
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:568)
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:190)
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:113)
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at com.google.inject.Guice.createInjector(Guice.java:87)
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at org.graylog2.shared.bindings.GuiceInjectorHolder.createInjector(GuiceInjectorHolder.java:34)
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at org.graylog2.bootstrap.CmdLineTool.setupInjector(CmdLineTool.java:490)
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:295)
Nov 18 15:16:32 ubuntu_1804 graylog-server[7629]:         at org.graylog2.bootstrap.Main.main(Main.java:45)

Graylog server log

2022-11-18T15:16:08.808+01:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2022-11-18T15:16:10.985+01:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 4.3.9 [org.graylog.aws.AWSPlugin]
2022-11-18T15:16:10.988+01:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integrations 4.3.9 [org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin]
2022-11-18T15:16:10.990+01:00 INFO  [CmdLineTool] Loaded plugin: Integrations 4.3.9 [org.graylog.integrations.IntegrationsPlugin]
2022-11-18T15:16:10.992+01:00 INFO  [CmdLineTool] Loaded plugin: Collector 4.3.9 [org.graylog.plugins.collector.CollectorPlugin]
2022-11-18T15:16:10.995+01:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise 4.3.9 [org.graylog.plugins.enterprise.EnterprisePlugin]
2022-11-18T15:16:10.998+01:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES6 Support) 4.3.9 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es6.EnterpriseES6Plugin]
2022-11-18T15:16:11.000+01:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES7 Support) 4.3.9 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es7.EnterpriseES7Plugin]
2022-11-18T15:16:11.003+01:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.3.9 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2022-11-18T15:16:11.004+01:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.3.9+e2c6648 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2022-11-18T15:16:11.005+01:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.3.9+e2c6648 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2022-11-18T15:16:11.109+01:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms2g -Xmx2g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2022-11-18T15:16:13.075+01:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2022-11-18T15:16:13.300+01:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-11-18T15:16:13.470+01:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:34}] to localhost:27017
2022-11-18T15:16:13.489+01:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 28]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=9325632}
2022-11-18T15:16:13.540+01:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:35}] to localhost:27017
2022-11-18T15:16:13.628+01:00 INFO  [connection] Closed connection [connectionId{localValue:2, serverValue:35}] to localhost:27017 because the pool has been closed.
2022-11-18T15:16:13.637+01:00 INFO  [MongoDBPreflightCheck] Connected to MongoDB version 4.0.28
2022-11-18T15:16:13.967+01:00 INFO  [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <Elasticsearch:7.10.2>
2022-11-18T15:16:14.291+01:00 INFO  [Version] HV000001: Hibernate Validator null
2022-11-18T15:16:25.596+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:25.651+01:00 INFO  [NodeId] Node ID: c251c81e-9729-4ec5-9348-7c7132fe8d8d
2022-11-18T15:16:26.202+01:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2022-11-18T15:16:26.206+01:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-11-18T15:16:26.214+01:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:36}] to localhost:27017
2022-11-18T15:16:26.217+01:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 28]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1509000}
2022-11-18T15:16:26.224+01:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:37}] to localhost:27017
2022-11-18T15:16:27.241+01:00 INFO  [ElasticsearchVersionProvider] Elasticsearch cluster is running Elasticsearch:7.10.2
2022-11-18T15:16:28.685+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:29.185+01:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:38}] to localhost:27017
2022-11-18T15:16:29.278+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:30.034+01:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-11-18T15:16:30.068+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:30.104+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:31.129+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:31.687+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:31.694+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:31.706+01:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-11-18T15:16:31.715+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:31.803+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:31.851+01:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-11-18T15:16:31.883+01:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): java.lang.NumberFormatException: For input string: ""
2022-11-18T15:16:31.883+01:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): java.lang.NumberFormatException: For input string: ""
2022-11-18T15:16:31.884+01:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): java.lang.NumberFormatException: For input string: ""
2022-11-18T15:16:31.884+01:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): java.lang.NumberFormatException: For input string: ""
2022-11-18T15:16:31.884+01:00 ERROR [CmdLineTool] Guice error (more detail on log level debug): java.lang.NumberFormatException: For input string: ""

/etc/default/graylog-server

# Path to the java executable.
JAVA=/usr/bin/java

# Default Java options for heap and garbage collection.
GRAYLOG_SERVER_JAVA_OPTS="-Xms2g -Xmx2g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow"

# Avoid endless loop with some TLSv1.3 implementations.
GRAYLOG_SERVER_JAVA_OPTS="$GRAYLOG_SERVER_JAVA_OPTS -Djdk.tls.acknowledgeCloseNotify=true"

# Fix for log4j CVE-2021-44228
GRAYLOG_SERVER_JAVA_OPTS="$GRAYLOG_SERVER_JAVA_OPTS -Dlog4j2.formatMsgNoLookups=true"

# Pass some extra args to graylog-server. (i.e. "-d" to enable debug mode)
#GRAYLOG_SERVER_ARGS=""

# Program that will be used to wrap the graylog-server command. Useful to
# support programs like authbind.
#GRAYLOG_COMMAND_WRAPPER=""

Thanks for you help!

gsmith · November 21, 2022, 11:48pm

Hello @dani

I’m assuming this is just one node with ES, GL & MongoDb on it? By chance do you have another volume attached to the node?

This line here.

 ERROR [CmdLineTool] Guice error (more detail on log level debug): java.lang.NumberFormatException: For input string: ""

Edit: Is that the only error you have in Graylog log file?
Since you stated JAVA what version are you using?
What trouble shooting have you done?
Sometimes it could be that the journal got corrupted, But I’m not seeing that error it in the log files. It shows “input string”.

EDIT:2
Doing some research on this, I’m wonder if it may have something to do with your inputs/Indices. Are you able to manually rotate your indices and restart Graylog service? Don’t for get to tail log file.

By the way pretty much everything runs on java in Graylog

The reason I stated this was from here “java.lang.NumberFormatException:”.

https://docs.oracle.com/javase/8/docs/api/java/lang/NumberFormatException.html

TBH since this server crashed, if you don’t mind losing some data/messages. Stop Graylog service clean out the journal then restart Graylog service again. I had to do this a couple time in the past. Tail the log file see if that clears things up.

dani · November 23, 2022, 5:09pm

Hello @gsmith

Yes, I have just one node with ES, GL & Mongo DB and no other volume.
Can’t find other errors. With JAVA I actually meant JVM arguments in this line:

2022-11-18T15:16:11.109+01:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms2g -Xmx2g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -Dlog4j2.formatMsgNoLookups=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb

Troubleshooting so far:

checking my configuration files
reverting changes for java heap sizes, as I had troubles with that previously
purged elasticsearch and made a new install (I upgraded once to elasticsearch 7.11 and it worked for a while - know I shouldn’t do that)
upgraded Graylog to 4.3

But your hint “Stop Graylog service clean out the journal then restart Graylog service again.” did it!!
Now I have some Elasticsearch errors to investigate, but I can log on to Graylog again.

Thanks for your help!

gsmith · November 23, 2022, 10:25pm

Hey @dani

awesome-yes-will-ferrell (1)

Glad I could help, If you could mark your solution resolved for future searches that would be great

system · December 7, 2022, 10:26pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
After upgrade from 4.0 to 4.1 graylog stop working Graylog Central (peer support)	6	1298	August 9, 2021
Graylog can't active Graylog Central (peer support)	6	3422	April 2, 2018
Graylog-server.service fails to start / no logs Graylog Central (peer support)	3	6627	January 6, 2021
Unable to start graylog Graylog Central (peer support)	17	2613	August 31, 2017
No access graylog webinterface Graylog Central (peer support)	1	330	July 2, 2019

Graylog service active but application not running

Related topics