Error in Server.log

Hello,

Has anyone seen this error before? It just keeps restart the graylog server service. And repeats this error message, like in a loop.

2020-10-16T14:33:24.662Z ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NumberFormatException: For input string: “H
U
V
T”
2020-10-16T14:33:24.663Z ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NumberFormatException: For input string: “H
U
V
T”
2020-10-16T14:33:24.664Z ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NumberFormatException: For input string: “H
U
V
T”
2020-10-16T14:33:24.664Z ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NumberFormatException: For input string: “H
U
V
T”
2020-10-16T14:33:24.665Z ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NumberFormatException: For input string: “H
U
V
T”
2020-10-16T14:33:24.665Z ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NumberFormatException: For input string: “H
U
V
T”

I have enabled debug with Shoothubs help and really did not add anything to the server.log file.

Here is what is in the middle of all of these errors, each time the Graylog server service attempts to restart.

020-10-16T14:33:19.740Z INFO [CmdLineTool] Loaded plugin: AWS plugins 3.3.7 [org.graylog.aws.AWSPlugin]
2020-10-16T14:33:19.743Z INFO [CmdLineTool] Loaded plugin: Enterprise Integrations 3.3.1 [org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin]
2020-10-16T14:33:19.744Z INFO [CmdLineTool] Loaded plugin: Integrations 3.3.1 [org.graylog.integrations.IntegrationsPlugin]
2020-10-16T14:33:19.745Z INFO [CmdLineTool] Loaded plugin: Collector 3.3.7 [org.graylog.plugins.collector.CollectorPlugin]
2020-10-16T14:33:19.746Z INFO [CmdLineTool] Loaded plugin: Graylog Enterprise 3.3.1 [org.graylog.plugins.enterprise.EnterprisePlugin]
2020-10-16T14:33:19.746Z INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 3.3.7 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2020-10-16T14:33:19.924Z INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2020-10-16T14:33:20.147Z INFO [Version] HV000001: Hibernate Validator null
2020-10-16T14:33:23.027Z INFO [InputBufferImpl] Message journal is enabled.
2020-10-16T14:33:23.046Z INFO [NodeId] Node ID: b5618c73-9928-4a85-bed5-6fda12070e67
2020-10-16T14:33:23.204Z INFO [InputBufferImpl] Message journal is enabled.
2020-10-16T14:33:23.284Z INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2020-10-16T14:33:23.331Z INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2020-10-16T14:33:23.359Z INFO [connection] Opened connection [connectionId{localValue:1, serverValue:450}] to localhost:27017
2020-10-16T14:33:23.364Z INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 19]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=3073238}
2020-10-16T14:33:23.376Z INFO [connection] Opened connection [connectionId{localValue:2, serverValue:451}] to localhost:27017
2020-10-16T14:33:23.570Z INFO [InputBufferImpl] Message journal is enabled.
2020-10-16T14:33:23.722Z INFO [AbstractJestClient] Setting server pool to a list of 1 servers: [http://127.0.0.1:9200]
2020-10-16T14:33:23.723Z INFO [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2020-10-16T14:33:23.793Z INFO [JestClientFactory] Using custom ObjectMapper instance
2020-10-16T14:33:23.794Z INFO [JestClientFactory] Node Discovery disabled…
2020-10-16T14:33:23.795Z INFO [JestClientFactory] Idle connection reaping disabled…
2020-10-16T14:33:24.260Z INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2020-10-16T14:33:24.415Z INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2020-10-16T14:33:24.420Z INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2020-10-16T14:33:24.430Z INFO [InputBufferImpl] Message journal is enabled.
2020-10-16T14:33:24.614Z INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2020-10-16T14:33:24.619Z INFO [InputBufferImpl] Message journal is enabled.
2020-10-16T14:33:24.649Z INFO [InputBufferImpl] Message journal is enabled.
2020-10-16T14:33:24.651Z INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy .

Been trying to find a solution to this issue with no luck. So thought someone might have already seen it or can determine from the logs as to what could be the issue and some ideas on how to fix it.

Thanks,

I ended up finding a fix to my issue, it was related to Kafka Journaling.
Here is what I saw in console on Graylog server startup:
at org.graylog2.shared.journal.KafkaJournal.(KafkaJournal.java:191)

Also, saw this message in the server.log file.
ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, java.lang.NumberFormatException: For input string: “H
U
V
T”

I came across this blog article and it helped me get Graylog up and running.

I changed back to the default Graylog Kafka path, once I was able to get Graylog up and running.

Step # 1

Followed steps in link.

Step # 2

Copy the file in /mnt.graylog to the original journaling directory as listed in server.conf file.

Step # 3

Perform a chown for that new directory to graylog:graylog

user@server-graylog-01:/var/lib/graylog-server/journal$ sudo chown -R graylog:graylog graylog-server/

You now should be back to normal.

Hope this helps,