No Messages In or Out after rebooting Graylog Server

Graylog Central (peer support)
1

Description of your problem

After rebooting the Graylog server no message are coming in or out.

Description of steps you’ve taken to attempt to solve the issue

I thought it might be a problem with iptables, but they are persistent after the reboot.

```
# Generated by iptables-save v1.6.1 on Wed Sep 22 09:29:43 2021
*nat
:PREROUTING ACCEPT [360:45924]
:INPUT ACCEPT [3:182]
:OUTPUT ACCEPT [57:3591]
:POSTROUTING ACCEPT [57:3591]
-A PREROUTING -p tcp -m tcp --dport 514 -j REDIRECT --to-ports 1514
-A PREROUTING -p udp -m udp --dport 514 -j REDIRECT --to-ports 1514
COMMIT
# Completed on Wed Sep 22 09:29:43 2021
# Generated by iptables-save v1.6.1 on Wed Sep 22 09:29:43 2021
*filter
:INPUT ACCEPT [6207039:4313946592]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2529722:1022112025]
COMMIT
# Completed on Wed Sep 22 09:29:43 2021
```

I’ve looked through the forums and nothing seems to be similar to what I’m experiencing.

I’m running Ubuntu 18.04 on Hyper-v. I’ll created a checkpoint prior to rebooting and if I roll back to that checkpoint Graylog works.

Environmental information

Operating system information

  • Hyper-V Server 2019
  • Ubuntu 18.04

Package versions

  • Graylog 4.2.0+5adccc3 on graylog (Private Build 1.8.0_292 on Linux 4.15.0-159-generic)
  • MongoDB v4.0.27
  • Elasticsearch 7.10.2
1 Like
2

Was it working before? What lead you to believe it was iptables? What changed after your hyper-v checkpoint other than a reboot? Was it an upgrade of Graylog or the OS or both? What are you seeing in your Graylog logs? You can watch them with:

tail -f /var/log/graylog-server/server.log

3

  • Was it working before?
    Yes

  • What lead you to believe it was iptables?
    I struggled with iptables when I installed Graylog. I also rebooted the server once before and got it working. I remember playing around with iptables, but I don’t think that is was what made it work in the end.

  • What changed after your hyper-v checkpoint other than a reboot?
    Nothing changed.

  • Was it an upgrade of Graylog or the OS or both?
    No upgrade to Graylog or the OS

  • Graylog logs
    The log file from the point of the reboot is to large to post here. This is what the forum will allow.


2021-11-16T16:38:29.545-05:00 WARN  [ClusterEventPeriodical] Error while reading cluster events from MongoDB, retrying.
com.mongodb.MongoQueryException: Query failed with error code 11600 and error message 'interrupted at shutdown' on server localhost:27017
        at com.mongodb.operation.FindOperation$1.call(FindOperation.java:735) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation$1.call(FindOperation.java:725) ~[graylog.jar:?]
        at com.mongodb.operation.OperationHelper.withReadConnectionSource(OperationHelper.java:463) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation.execute(FindOperation.java:725) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation.execute(FindOperation.java:89) ~[graylog.jar:?]
        at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:196) ~[graylog.jar:?]
        at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:177) ~[graylog.jar:?]
        at com.mongodb.DBCursor.initializeCursor(DBCursor.java:989) ~[graylog.jar:?]
        at com.mongodb.DBCursor.hasNext(DBCursor.java:172) ~[graylog.jar:?]
        at org.mongojack.DBCursor.hasNext(DBCursor.java:330) ~[graylog.jar:?]
        at org.graylog2.events.ClusterEventPeriodical.doRun(ClusterEventPeriodical.java:152) [graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
2021-11-16T16:38:29.545-05:00 ERROR [NodePingThread] Uncaught exception in periodical
com.mongodb.MongoQueryException: Query failed with error code 11600 and error message 'interrupted at shutdown' on server localhost:27017
        at com.mongodb.operation.FindOperation$1.call(FindOperation.java:735) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation$1.call(FindOperation.java:725) ~[graylog.jar:?]
        at com.mongodb.operation.OperationHelper.withReadConnectionSource(OperationHelper.java:463) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation.execute(FindOperation.java:725) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation.execute(FindOperation.java:89) ~[graylog.jar:?]
        at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:196) ~[graylog.jar:?]
        at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:177) ~[graylog.jar:?]
        at com.mongodb.DBCursor.initializeCursor(DBCursor.java:989) ~[graylog.jar:?]
        at com.mongodb.DBCursor.hasNext(DBCursor.java:172) ~[graylog.jar:?]
        at com.mongodb.DBCursor.one(DBCursor.java:790) ~[graylog.jar:?]
        at com.mongodb.DBCollection.findOne(DBCollection.java:867) ~[graylog.jar:?]
        at com.mongodb.DBCollection.findOne(DBCollection.java:827) ~[graylog.jar:?]
        at com.mongodb.DBCollection.findOne(DBCollection.java:770) ~[graylog.jar:?]
        at org.graylog2.database.PersistedServiceImpl.findOne(PersistedServiceImpl.java:128) ~[graylog.jar:?]
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:73) ~[graylog.jar:?]
        at org.graylog2.cluster.NodeServiceImpl.byNodeId(NodeServiceImpl.java:84) ~[graylog.jar:?]
        at org.graylog2.periodical.NodePingThread.doRun(NodePingThread.java:62) ~[graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
2021-11-16T16:38:29.545-05:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-11-16T16:38:29.546-05:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-11-16T16:38:29.544-05:00 ERROR [AWSInstanceNameLookupProcessor] Could not refresh AWS instance lookup table.
java.util.concurrent.ExecutionException: com.mongodb.MongoQueryException: Query failed with error code 11600 and error message 'interrupted at shutdown' on server localhost:27017
        at com.github.rholder.retry.Retryer$ExceptionAttempt.<init>(Retryer.java:254) ~[graylog.jar:?]
        at com.github.rholder.retry.Retryer.call(Retryer.java:163) ~[graylog.jar:?]
        at org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor.waitForMigrationCompletion(AWSInstanceNameLookupProcessor.java:136) ~[graylog-plugin-aws-4.2.0.jar:?]
        at org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor.access$000(AWSInstanceNameLookupProcessor.java:42) ~[graylog-plugin-aws-4.2.0.jar:?]
        at org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor$1.run(AWSInstanceNameLookupProcessor.java:82) [graylog-plugin-aws-4.2.0.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Caused by: com.mongodb.MongoQueryException: Query failed with error code 11600 and error message 'interrupted at shutdown' on server localhost:27017
        at com.mongodb.operation.FindOperation$1.call(FindOperation.java:735) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation$1.call(FindOperation.java:725) ~[graylog.jar:?]
        at com.mongodb.operation.OperationHelper.withReadConnectionSource(OperationHelper.java:463) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation.execute(FindOperation.java:725) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation.execute(FindOperation.java:89) ~[graylog.jar:?]
        at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:196) ~[graylog.jar:?]
        at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:177) ~[graylog.jar:?]
        at com.mongodb.DBCursor.initializeCursor(DBCursor.java:989) ~[graylog.jar:?]
        at com.mongodb.DBCursor.hasNext(DBCursor.java:172) ~[graylog.jar:?]
        at org.mongojack.DBCursor.hasNext(DBCursor.java:330) ~[graylog.jar:?]
        at org.mongojack.JacksonDBCollection.findOne(JacksonDBCollection.java:1408) ~[graylog.jar:?]
        at org.mongojack.JacksonDBCollection.findOne(JacksonDBCollection.java:1369) ~[graylog.jar:?]
        at org.mongojack.JacksonDBCollection.findOne(JacksonDBCollection.java:1343) ~[graylog.jar:?]
        at org.graylog2.cluster.ClusterConfigServiceImpl.get(ClusterConfigServiceImpl.java:102) ~[graylog.jar:?]
        at org.graylog2.cluster.ClusterConfigServiceImpl.get(ClusterConfigServiceImpl.java:119) ~[graylog.jar:?]
        at org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor.lambda$waitForMigrationCompletion$1(AWSInstanceNameLookupProcessor.java:136) ~[graylog-plugin-aws-4.2.0.jar:?]
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[graylog.jar:?]
        at com.github.rholder.retry.Retryer.call(Retryer.java:160) ~[graylog.jar:?]
        ... 10 more
2021-11-16T16:38:29.546-05:00 ERROR [AWSInstanceNameLookupProcessor] Could not refresh AWS instance lookup table.
java.util.concurrent.ExecutionException: com.mongodb.MongoQueryException: Query failed with error code 11600 and error message 'interrupted at shutdown' on server localhost:27017
        at com.github.rholder.retry.Retryer$ExceptionAttempt.<init>(Retryer.java:254) ~[graylog.jar:?]
        at com.github.rholder.retry.Retryer.call(Retryer.java:163) ~[graylog.jar:?]
        at org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor.waitForMigrationCompletion(AWSInstanceNameLookupProcessor.java:136) ~[graylog-plugin-aws-4.2.0.jar:?]
        at org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor.access$000(AWSInstanceNameLookupProcessor.java:42) ~[graylog-plugin-aws-4.2.0.jar:?]
        at org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor$1.run(AWSInstanceNameLookupProcessor.java:82) [graylog-plugin-aws-4.2.0.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Caused by: com.mongodb.MongoQueryException: Query failed with error code 11600 and error message 'interrupted at shutdown' on server localhost:27017
        at com.mongodb.operation.FindOperation$1.call(FindOperation.java:735) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation$1.call(FindOperation.java:725) ~[graylog.jar:?]
        at com.mongodb.operation.OperationHelper.withReadConnectionSource(OperationHelper.java:463) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation.execute(FindOperation.java:725) ~[graylog.jar:?]
        at com.mongodb.operation.FindOperation.execute(FindOperation.java:89) ~[graylog.jar:?]
        at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:196) ~[graylog.jar:?]
        at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:177) ~[graylog.jar:?]
        at com.mongodb.DBCursor.initializeCursor(DBCursor.java:989) ~[graylog.jar:?]
        at com.mongodb.DBCursor.hasNext(DBCursor.java:172) ~[graylog.jar:?]
        at org.mongojack.DBCursor.hasNext(DBCursor.java:330) ~[graylog.jar:?]
        at org.mongojack.JacksonDBCollection.findOne(JacksonDBCollection.java:1408) ~[graylog.jar:?]
        at org.mongojack.JacksonDBCollection.findOne(JacksonDBCollection.java:1369) ~[graylog.jar:?]
        at org.mongojack.JacksonDBCollection.findOne(JacksonDBCollection.java:1343) ~[graylog.jar:?]
        at org.graylog2.cluster.ClusterConfigServiceImpl.get(ClusterConfigServiceImpl.java:102) ~[graylog.jar:?]
        at org.graylog2.cluster.ClusterConfigServiceImpl.get(ClusterConfigServiceImpl.java:119) ~[graylog.jar:?]
        at org.graylog.aws.processors.instancelookup.AWSInstanceNameLookupProcessor.lambda$waitForMigrationCompletion$1(AWSInstanceNameLookupProcessor.java:136) ~[graylog-plugin-aws-4.2.0.jar:?]
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78) ~[graylog.jar:?]
        at com.github.rholder.retry.Retryer.call(Retryer.java:160) ~[graylog.jar:?]
        ... 10 more
2021-11-16T16:39:15.420-05:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2021-11-16T16:39:19.913-05:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 4.2.0 [org.graylog.aws.AWSPlugin]
2021-11-16T16:39:19.915-05:00 INFO  [CmdLineTool] Loaded plugin: Integrations 4.1.5 [org.graylog.integrations.IntegrationsPlugin]
2021-11-16T16:39:19.916-05:00 INFO  [CmdLineTool] Loaded plugin: Collector 4.2.0 [org.graylog.plugins.collector.CollectorPlugin]
2021-11-16T16:39:19.917-05:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.2.0 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-11-16T16:39:19.918-05:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.2.0+5adccc3 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2021-11-16T16:39:19.918-05:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.2.0+5adccc3 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2021-11-16T16:39:20.948-05:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-11-16T16:39:22.159-05:00 INFO  [Version] HV000001: Hibernate Validator null
2021-11-16T16:39:35.939-05:00 INFO  [InputBufferImpl] Message journal is enabled.
2021-11-16T16:39:36.023-05:00 INFO  [NodeId] Node ID: 0646dbed-0a28-49e5-bf71-00e9e67fcfd9
2021-11-16T16:39:37.057-05:00 INFO  [LogManager] Loading logs.
2021-11-16T16:39:37.150-05:00 WARN  [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000000252596337.index, deleting and rebuilding index...
2021-11-16T16:39:38.285-05:00 INFO  [LogManager] Logs loading complete.
2021-11-16T16:39:38.290-05:00 INFO  [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-11-16T16:39:38.776-05:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2021-11-16T16:39:39.011-05:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-11-16T16:39:39.129-05:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:1}] to localhost:27017
2021-11-16T16:39:39.142-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 27]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=5049514}
2021-11-16T16:39:39.208-05:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:2}] to localhost:27017
2021-11-16T16:39:40.358-05:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2021-11-16T16:39:41.996-05:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2021-11-16T16:39:47.000-05:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2021-11-16T16:39:52.604-05:00 INFO  [ElasticsearchVersionProvider] Elasticsearch cluster is running v7.10.2
2021-11-16T16:39:55.570-05:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-11-16T16:39:55.812-05:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:3}] to localhost:27017
2021-11-16T16:39:57.244-05:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-11-16T16:40:00.977-05:00 INFO  [ServerBootstrap] Graylog server 4.2.0+5adccc3 starting up
2021-11-16T16:40:00.978-05:00 INFO  [ServerBootstrap] JRE: Private Build 1.8.0_292 on Linux 4.15.0-162-generic
2021-11-16T16:40:00.979-05:00 INFO  [ServerBootstrap] Deployment: deb
2021-11-16T16:40:00.986-05:00 INFO  [ServerBootstrap] OS: Ubuntu 18.04.6 LTS (bionic)
2021-11-16T16:40:00.986-05:00 INFO  [ServerBootstrap] Arch: amd64
2021-11-16T16:40:01.125-05:00 INFO  [PeriodicalsService] Starting 29 periodicals ...
2021-11-16T16:40:01.139-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2021-11-16T16:40:01.158-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2021-11-16T16:40:01.167-05:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2021-11-16T16:40:01.167-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2021-11-16T16:40:01.188-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2021-11-16T16:40:01.207-05:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2021-11-16T16:40:01.219-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2021-11-16T16:40:01.269-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2021-11-16T16:40:01.278-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2021-11-16T16:40:01.279-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2021-11-16T16:40:01.286-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2021-11-16T16:40:01.303-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2021-11-16T16:40:01.304-05:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2021-11-16T16:40:01.305-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2021-11-16T16:40:01.306-05:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2021-11-16T16:40:01.306-05:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2021-11-16T16:40:01.306-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2021-11-16T16:40:01.315-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2021-11-16T16:40:01.336-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2021-11-16T16:40:01.415-05:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:4}] to localhost:27017
2021-11-16T16:40:01.417-05:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:5}] to localhost:27017
2021-11-16T16:40:01.448-05:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:8}] to localhost:27017
2021-11-16T16:40:01.452-05:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:7}] to localhost:27017
2021-11-16T16:40:01.475-05:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:6}] to localhost:27017
2021-11-16T16:40:01.509-05:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2021-11-16T16:40:01.509-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2021-11-16T16:40:01.510-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2021-11-16T16:40:01.510-05:00 INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2021-11-16T16:40:01.483-05:00 INFO  [connection] Opened connection [connectionId{localValue:9, serverValue:9}] to localhost:27017
2021-11-16T16:40:01.515-05:00 INFO  [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2021-11-16T16:40:01.516-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2021-11-16T16:40:01.517-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2021-11-16T16:40:01.517-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2021-11-16T16:40:01.543-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2021-11-16T16:40:01.546-05:00 INFO  [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2021-11-16T16:40:01.546-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2021-11-16T16:40:01.667-05:00 INFO  [LookupTableService] Data Adapter geoip/614e28029f8bf82a3736d378 [@3cbb6a91] STARTING
2021-11-16T16:40:01.735-05:00 INFO  [LookupTableService] Data Adapter geoip/614e28029f8bf82a3736d378 [@3cbb6a91] RUNNING
2021-11-16T16:40:01.736-05:00 INFO  [LookupDataAdapterRefreshService] Adding job for <geoip/614e28029f8bf82a3736d378/@3cbb6a91> [interval=60000ms]
2021-11-16T16:40:02.295-05:00 INFO  [LookupTableService] Cache geoip/614e28469f8bf82a3736d3c2 [@725b0b0a] STARTING
2021-11-16T16:40:02.316-05:00 INFO  [LookupTableService] Cache geoip/614e28469f8bf82a3736d3c2 [@725b0b0a] RUNNING
2021-11-16T16:40:02.403-05:00 INFO  [LookupTableService] Starting lookup table geoip/614e288a9f8bf82a3736d40e [@2375d4d0] using cache geoip/614e28469f8bf82a3736d3c2 [@725b0b0a], data adapter geoip/614e28029f8bf82a3736d378 [@3cbb6a91]
2021-11-16T16:40:03.322-05:00 INFO  [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2021-11-16T16:40:10.214-05:00 INFO  [NetworkListener] Started listener bound to [172.16.2.15:9000]
2021-11-16T16:40:10.217-05:00 INFO  [HttpServer] [HttpServer] Started.
2021-11-16T16:40:10.217-05:00 INFO  [JerseyService] Started REST API at <172.16.2.15:9000>
2021-11-16T16:40:10.219-05:00 INFO  [ServerBootstrap] Services started, startup times in ms: {ConfigurationEtagService [RUNNING]=174, OutputSetupService [RUNNING]=179, BufferSynchronizerService [RUNNING]=179, PrometheusExporter [RUNNING]=180, JobSchedulerService [RUNNING]=189, EtagService [RUNNING]=207, InputSetupService [RUNNING]=264, LocalKafkaMessageQueueWriter [RUNNING]=266, LocalKafkaMessageQueueReader [RUNNING]=266, FailureHandlingService [RUNNING]=266, GracefulShutdownService [RUNNING]=267, UserSessionTerminationService [RUNNING]=271, UrlWhitelistService [RUNNING]=292, LocalKafkaJournal [RUNNING]=296, MongoDBProcessingStatusRecorderService [RUNNING]=322, PeriodicalsService [RUNNING]=439, StreamCacheService [RUNNING]=440, LookupTableService [RUNNING]=1204, JerseyService [RUNNING]=9109}
2021-11-16T16:40:10.225-05:00 INFO  [ServiceManagerListener] Services are healthy
2021-11-16T16:40:10.233-05:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2021-11-16T16:40:10.369-05:00 INFO  [ServerBootstrap] Graylog server up and running.
2021-11-16T16:40:10.369-05:00 INFO  [InputStateListener] Input [Syslog UDP/614b32dd9f8bf82a37339ca9] is now STARTING
2021-11-16T16:40:10.376-05:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/614b86e49f8bf82a3733f849] is now STARTING
2021-11-16T16:40:11.119-05:00 WARN  [Bootstrap] Unknown channel option 'io.netty.channel.unix.UnixChannelOption#SO_REUSEPORT' for channel '[id: 0xdebf0307]'
2021-11-16T16:40:11.151-05:00 WARN  [Bootstrap] Unknown channel option 'io.netty.channel.unix.UnixChannelOption#SO_REUSEPORT' for channel '[id: 0xfeed0eba]'
2021-11-16T16:40:11.270-05:00 INFO  [InputStateListener] Input [Syslog UDP/614b32dd9f8bf82a37339ca9] is now RUNNING
2021-11-16T16:40:11.271-05:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=Local graylog, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=0646dbed-0a28-49e5-bf71-00e9e67fcfd9} (channel [id: 0xdebf0307, L:/0:0:0:0:0:0:0:0:1514]) should be >= 262144 but is 212992.
2021-11-16T16:40:11.268-05:00 WARN  [UdpTransport] Failed to start channel for input RawUDPInput{title=FortiGate, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=0646dbed-0a28-49e5-bf71-00e9e67fcfd9}
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method) ~[?:1.8.0_292]
        at sun.nio.ch.Net.bind(Net.java:461) ~[?:1.8.0_292]
        at sun.nio.ch.DatagramChannelImpl.bind(DatagramChannelImpl.java:698) ~[?:1.8.0_292]
        at io.netty.util.internal.SocketUtils$6.run(SocketUtils.java:133) ~[graylog.jar:?]
        at io.netty.util.internal.SocketUtils$6.run(SocketUtils.java:130) ~[graylog.jar:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_292]
        at io.netty.util.internal.SocketUtils.bind(SocketUtils.java:130) ~[graylog.jar:?]
        at io.netty.channel.socket.nio.NioDatagramChannel.doBind0(NioDatagramChannel.java:200) ~[graylog.jar:?]
        at io.netty.channel.socket.nio.NioDatagramChannel.doBind(NioDatagramChannel.java:195) ~[graylog.jar:?]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:550) [graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334) [graylog.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506) [graylog.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491) [graylog.jar:?]
        at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973) [graylog.jar:?]
        at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:248) [graylog.jar:?]
        at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356) [graylog.jar:?]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [graylog.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [graylog.jar:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [graylog.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [graylog.jar:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
2021-11-16T16:40:11.292-05:00 ERROR [InputLauncher] The [org.graylog2.inputs.raw.udp.RawUDPInput] input with ID <614b86e49f8bf82a3733f849> misfired. Reason: Address already in use.
org.graylog2.plugin.inputs.MisfireException: org.graylog2.plugin.inputs.MisfireException: java.net.BindException: Address already in use
        at org.graylog2.plugin.inputs.MessageInput.launch(MessageInput.java:158) ~[graylog.jar:?]
        at org.graylog2.shared.inputs.InputLauncher$1.run(InputLauncher.java:84) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]

2021-11-16T16:40:19.106-05:00 WARN  [LookupTableService] Lookup table <geoip-lookup> does not exist
4

Right form the start your Gralog MongoDB looks unhappy. Check Mongodb and elasticsearch services - are they running? Since that is all your Graylog settings, not much else will be happy either… Start by makeing sure MongoDB is running/happy…

sudo systemctl status mongod
sudo systemctl status elasticsearch

5

Hello,

Just chiming in. I looked over your log file real quick.

I believe error 11600 means that the client is trying to do an operation on a server that is shutting down. So, if you provide the driver with a connection URI the driver should reconnect as soon as it’s available. You may want to look here.

This may have to do with your reboot, You normally receive this error when Graylog starts first and elasticsearch has not fully started OR your elasticsearch service failed to start.

Seams like you have a issue on a input.

Best suggestion I could tell ya is execute what @tmacgbay suggested first. Make sure Elasticsearch is started before the other services (MongoDb, Graylog). If there is no problems then start MongoDb and check for errors/warnings. Then start Graylog service and either tail the log files while its starting up or check status of the service/s to see if there are any issues.

Make sure that the service are enabled. So, after a reboot these service will start back up.

 sudo systemctl enable graylog-server
 sudo systemctl enable mongod
 sudo systemctl enable elasticsearch

Hope that helps

6

Thanks @gsmith and @tmacgbay.

I rolled the server back to a working state and ran systemctl status for mongod, elasticsearch and graylog. Here is the status before and after the restart.

Before restart when Graylog is collecting messages:


● mongod.service - MongoDB Database Server
   Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-10-13 10:07:51 EDT; 1 months 4 days ago
     Docs: https://docs.mongodb.org/manual
 Main PID: 1133 (mongod)
   CGroup: /system.slice/mongod.service
           └─1133 /usr/bin/mongod --config /etc/mongod.conf

Oct 13 10:07:51 graylog systemd[1]: Started MongoDB Database Server.
ldog@graylog:~$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-10-13 10:08:39 EDT; 1 months 4 days ago
     Docs: https://www.elastic.co
 Main PID: 1099 (java)
    Tasks: 62 (limit: 4591)
   CGroup: /system.slice/elasticsearch.service
           └─1099 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTo

Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at org.elasticsearch.action.search.AbstractSearchAsyncAction.raisePhaseFailure(AbstractSearchAsyncActi
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:100)
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(Thre
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at java.base/java.lang.Thread.run(Thread.java:832)


ldog@graylog:~$ sudo systemctl status graylog-server.service
● graylog-server.service - Graylog server
   Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2021-10-14 15:11:59 EDT; 1 months 3 days ago
     Docs: http://docs.graylog.org/
 Main PID: 12199 (graylog-server)
    Tasks: 201 (limit: 4591)
   CGroup: /system.slice/graylog-server.service
           ├─12199 /bin/sh /usr/share/graylog-server/bin/graylog-server
           └─12253 /usr/bin/java -Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+

Oct 14 15:11:59 graylog systemd[1]: Started Graylog server.
Oct 14 15:14:00 graylog graylog-server[12199]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
Oct 14 15:14:00 graylog graylog-server[12199]: SLF4J: Defaulting to no-operation (NOP) logger implementation
Oct 14 15:14:00 graylog graylog-server[12199]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

After restart. Graylog is not collected messages:

ldog@graylog:~$ sudo systemctl status mongod
● mongod.service - MongoDB Database Server
   Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-11-17 08:59:18 EST; 5min ago
     Docs: https://docs.mongodb.org/manual
 Main PID: 1278 (mongod)
   CGroup: /system.slice/mongod.service
           └─1278 /usr/bin/mongod --config /etc/mongod.conf

Nov 17 08:59:18 graylog systemd[1]: Started MongoDB Database Server.
ldog@graylog:~$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-11-17 09:00:12 EST; 4min 56s ago
     Docs: https://www.elastic.co
 Main PID: 1103 (java)
    Tasks: 62 (limit: 4591)
   CGroup: /system.slice/elasticsearch.service
           └─1103 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTo

Nov 17 08:59:17 graylog systemd[1]: Starting Elasticsearch...
Nov 17 09:00:12 graylog systemd[1]: Started Elasticsearch.

ldog@graylog:~$ sudo systemctl status graylog-server.service
● graylog-server.service - Graylog server
   Loaded: loaded (/usr/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-11-17 08:59:18 EST; 5min ago
     Docs: http://docs.graylog.org/
 Main PID: 1243 (graylog-server)
    Tasks: 175 (limit: 4591)
   CGroup: /system.slice/graylog-server.service
           ├─1243 /bin/sh /usr/share/graylog-server/bin/graylog-server
           └─1723 /usr/bin/java -Xms1g -Xmx1g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+U

Nov 17 08:59:18 graylog systemd[1]: Started Graylog server.
7

Strange that your service status for elastic/mongo/graylog are cleaner when you are in the non-working state… I would have expected the reverse if anything.

Still… the first thing on our list is to look at Mongo since it is throwing the first errors. Check out Mongo log at:

/var/log/mongodb/mongod.log

You may want to have a second session to your Graylog server going and watch the logs as Mongo/Graylog comes up:

tail /var/log/mongodb/mongod.log -f

You can manually bring down graylog, then mongodb, then set log to observe Mongo log… start mongo, then start graylog-server… unless you see an issue in mongo that need to be dealt with first.

:thinking:… did you grab a snapshot of your Graylog server while it was running or were you shut down… the Mongo error talks about being in a starting or stopping state…

8

I just noticed that Ubuntu is saying:

*** System restart required ***

According to /var/run/reboot-required.pkgs

Linux-base
Linux-base

require a reboot. Would that be messing things up?

  • Graylog snapshot
    I’m taking the snapshot while the server and applications are running.

I’ll check out the log files in a minute.

9

I rebooted the server from a working snapshot. As before, no message in or out.

I manually stopped Elasticsearch, Mongo and Graylog.

Then I started each service and watched the MongoDB and Graylog logs.

  1. Elasticsearch
  2. MongoDB
  3. Graylog

There was nothing in the logs that screamed error at me. But there where a lot of them.

MongoDB:

2021-11-17T16:10:31.099-0500 I STORAGE  [initandlisten] WiredTiger message [1637183431:99627][1103:0x7f3061495a40], txn-recover: Set global recovery timestamp: 0
2021-11-17T16:10:31.424-0500 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(0, 0)
2021-11-17T16:10:31.472-0500 I STORAGE  [initandlisten] Starting to check the table logging settings for existing WiredTiger tables
2021-11-17T16:10:31.743-0500 I CONTROL  [initandlisten]
2021-11-17T16:10:31.743-0500 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2021-11-17T16:10:31.743-0500 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2021-11-17T16:10:31.744-0500 I CONTROL  [initandlisten]
2021-11-17T16:10:32.232-0500 I STORAGE  [initandlisten] Finished adjusting the table logging settings for existing WiredTiger tables
2021-11-17T16:10:32.517-0500 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'
2021-11-17T16:10:32.733-0500 I NETWORK  [initandlisten] waiting for connections on port 27017
2021-11-17T16:11:03.904-0500 I NETWORK  [listener] connection accepted from 127.0.0.1:57464 #1 (1 connection now open)
2021-11-17T16:11:03.930-0500 I NETWORK  [conn1] received client metadata from 127.0.0.1:57464 conn1: { driver: { name: "mongo-java-driver|legacy", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.15.0-162-generic" }, platform: "Java/Private Build/1.8.0_292-8u292-b10-0ubuntu1~18.04-b10" }
2021-11-17T16:11:04.058-0500 I NETWORK  [listener] connection accepted from 127.0.0.1:57466 #2 (2 connections now open)
2021-11-17T16:11:04.062-0500 I NETWORK  [conn2] received client metadata from 127.0.0.1:57466 conn2: { driver: { name: "mongo-java-driver|legacy", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.15.0-162-generic" }, platform: "Java/Private Build/1.8.0_292-8u292-b10-0ubuntu1~18.04-b10" }
2021-11-17T16:11:22.851-0500 I NETWORK  [listener] connection accepted from 127.0.0.1:57494 #3 (3 connections now open)
2021-11-17T16:11:22.863-0500 I NETWORK  [conn3] received client metadata from 127.0.0.1:57494 conn3: { driver: { name: "mongo-java-driver|legacy", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.15.0-162-generic" }, platform: "Java/Private Build/1.8.0_292-8u292-b10-0ubuntu1~18.04-b10" }
2021-11-17T16:11:26.281-0500 I NETWORK  [listener] connection accepted from 127.0.0.1:57496 #4 (4 connections now open)
2021-11-17T16:11:26.281-0500 I NETWORK  [conn4] received client metadata from 127.0.0.1:57496 conn4: { driver: { name: "mongo-java-driver|legacy", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.15.0-162-generic" }, platform: "Java/Private Build/1.8.0_292-8u292-b10-0ubuntu1~18.04-b10" }
2021-11-17T16:11:26.982-0500 I NETWORK  [listener] connection accepted from 127.0.0.1:57498 #5 (5 connections now open)
2021-11-17T16:11:26.983-0500 I NETWORK  [conn5] received client metadata from 127.0.0.1:57498 conn5: { driver: { name: "mongo-java-driver|legacy", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.15.0-162-generic" }, platform: "Java/Private Build/1.8.0_292-8u292-b10-0ubuntu1~18.04-b10" }
2021-11-17T16:11:26.985-0500 I NETWORK  [listener] connection accepted from 127.0.0.1:57500 #6 (6 connections now open)
2021-11-17T16:11:26.985-0500 I NETWORK  [listener] connection accepted from 127.0.0.1:57502 #7 (7 connections now open)
2021-11-17T16:11:26.985-0500 I NETWORK  [conn6] received client metadata from 127.0.0.1:57500 conn6: { driver: { name: "mongo-java-driver|legacy", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.15.0-162-generic" }, platform: "Java/Private Build/1.8.0_292-8u292-b10-0ubuntu1~18.04-b10" }
2021-11-17T16:11:26.985-0500 I NETWORK  [conn7] received client metadata from 127.0.0.1:57502 conn7: { driver: { name: "mongo-java-driver|legacy", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.15.0-162-generic" }, platform: "Java/Private Build/1.8.0_292-8u292-b10-0ubuntu1~18.04-b10" }
2021-11-17T16:11:26.992-0500 I NETWORK  [listener] connection accepted from 127.0.0.1:57504 #8 (8 connections now open)
2021-11-17T16:11:26.995-0500 I NETWORK  [conn8] received client metadata from 127.0.0.1:57504 conn8: { driver: { name: "mongo-java-driver|legacy", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.15.0-162-generic" }, platform: "Java/Private Build/1.8.0_292-8u292-b10-0ubuntu1~18.04-b10" }
2021-11-17T16:11:26.996-0500 I NETWORK  [listener] connection accepted from 127.0.0.1:57506 #9 (9 connections now open)
2021-11-17T16:11:26.997-0500 I NETWORK  [conn9] received client metadata from 127.0.0.1:57506 conn9: { driver: { name: "mongo-java-driver|legacy", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.15.0-162-generic" }, platform: "Java/Private Build/1.8.0_292-8u292-b10-0ubuntu1~18.04-b10" }
2021-11-17T16:11:27.010-0500 I NETWORK  [listener] connection accepted from 127.0.0.1:57508 #10 (10 connections now open)
2021-11-17T16:11:27.010-0500 I NETWORK  [conn10] received client metadata from 127.0.0.1:57508 conn10: { driver: { name: "mongo-java-driver|legacy", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.15.0-162-generic" }, platform: "Java/Private Build/1.8.0_292-8u292-b10-0ubuntu1~18.04-b10" }
2021-11-17T16:11:35.729-0500 I WRITE    [conn2] update graylog.system_messages command: { q: { _id: ObjectId('61957007bd759a65b4f2bf85') }, u: { caller: "org.graylog2.bootstrap.Main", content: "Started up.", timestamp: new Date(1637183495616), node_id: "0646dbed-0a28-49e5-bf71-00e9e67fcfd9", _id: ObjectId('61957007bd759a65b4f2bf85') }, multi: false, upsert: true } planSummary: IDHACK keysExamined:0 docsExamined:0 nMatched:0 nModified:0 upsert:1 keysInserted:2 numYields:1 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } } } storage:{ data: { bytesRead: 33510, timeReadingMicros: 56069 }, timeWaitingMicros: { schemaLock: 30454 } } 107ms
2021-11-17T16:11:35.730-0500 I COMMAND  [conn2] command graylog.$cmd command: update { update: "system_messages", ordered: true, $db: "graylog" } numYields:0 reslen:111 locks:{ Global: { acquireCount: { r: 4, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } } } storage:{} protocol:op_msg 108ms
2021-11-17T16:11:35.844-0500 I COMMAND  [conn5] command graylog.scheduler_triggers command: findAndModify { findAndModify: "scheduler_triggers", query: { $and: [ { lock.owner: null }, { status: "runnable" }, { start_time: { $lte: new Date(1637183495712) } }, { $or: [ { end_time: { $exists: false } }, { end_time: null }, { end_time: { $gt: new Date(1637183495712) } } ] }, { next_time: { $lte: new Date(1637183495712) } } ] }, sort: { next_time: 1 }, new: true, update: { $set: { lock.last_lock_time: new Date(1637183495712), triggered_at: new Date(1637183495712), lock.owner: "0646dbed-0a28-49e5-bf71-00e9e67fcfd9", status: "running" } }, $db: "graylog" } planSummary: IXSCAN { next_time: 1 } keysExamined:1 docsExamined:1 fromMultiPlanner:1 nMatched:1 nModified:1 keysInserted:2 keysDeleted:2 numYields:1 reslen:576 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } } } storage:{ data: { bytesRead: 1101, timeReadingMicros: 60262 }, timeWaitingMicros: { schemaLock: 11515 } } protocol:op_msg 128ms
2021-11-17T16:12:04.911-0500 I NETWORK  [listener] connection accepted from 127.0.0.1:57524 #11 (11 connections now open)
2021-11-17T16:12:04.912-0500 I NETWORK  [conn11] received client metadata from 127.0.0.1:57524 conn11: { driver: { name: "mongo-java-driver|legacy", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.15.0-162-generic" }, platform: "Java/Private Build/1.8.0_292-8u292-b10-0ubuntu1~18.04-b10" }
2021-11-17T16:12:04.965-0500 I NETWORK  [listener] connection accepted from 127.0.0.1:57526 #12 (12 connections now open)
2021-11-17T16:12:05.010-0500 I NETWORK  [conn12] received client metadata from 127.0.0.1:57526 conn12: { driver: { name: "mongo-java-driver|legacy", version: "3.12.1" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "4.15.0-162-generic" }, platform: "Java/Private Build/1.8.0_292-8u292-b10-0ubuntu1~18.04-b10" }

Graylog:

2021-11-17T16:11:03.572-05:00 INFO  [LogManager] Logs loading complete.
2021-11-17T16:11:03.582-05:00 INFO  [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-11-17T16:11:03.755-05:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2021-11-17T16:11:03.896-05:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-11-17T16:11:03.992-05:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:1}] to localhost:27017
2021-11-17T16:11:04.012-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 27]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=18218372}
2021-11-17T16:11:04.171-05:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:2}] to localhost:27017
2021-11-17T16:11:05.190-05:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2021-11-17T16:11:07.769-05:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2021-11-17T16:11:12.786-05:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2021-11-17T16:11:18.427-05:00 INFO  [ElasticsearchVersionProvider] Elasticsearch cluster is running v7.10.2
2021-11-17T16:11:21.220-05:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-11-17T16:11:22.794-05:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-11-17T16:11:22.865-05:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:3}] to localhost:27017
2021-11-17T16:11:26.030-05:00 INFO  [ServerBootstrap] Graylog server 4.2.0+5adccc3 starting up
2021-11-17T16:11:26.045-05:00 INFO  [ServerBootstrap] JRE: Private Build 1.8.0_292 on Linux 4.15.0-162-generic
2021-11-17T16:11:26.048-05:00 INFO  [ServerBootstrap] Deployment: deb
2021-11-17T16:11:26.049-05:00 INFO  [ServerBootstrap] OS: Ubuntu 18.04.6 LTS (bionic)
2021-11-17T16:11:26.049-05:00 INFO  [ServerBootstrap] Arch: amd64
2021-11-17T16:11:26.168-05:00 INFO  [PeriodicalsService] Starting 29 periodicals ...
2021-11-17T16:11:26.173-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2021-11-17T16:11:26.222-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2021-11-17T16:11:26.224-05:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2021-11-17T16:11:26.226-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2021-11-17T16:11:26.241-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2021-11-17T16:11:26.244-05:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2021-11-17T16:11:26.244-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2021-11-17T16:11:26.246-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2021-11-17T16:11:26.248-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2021-11-17T16:11:26.251-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2021-11-17T16:11:26.256-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2021-11-17T16:11:26.257-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2021-11-17T16:11:26.258-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2021-11-17T16:11:26.259-05:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2021-11-17T16:11:26.259-05:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2021-11-17T16:11:26.260-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2021-11-17T16:11:26.261-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2021-11-17T16:11:26.262-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2021-11-17T16:11:26.282-05:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:4}] to localhost:27017
2021-11-17T16:11:26.947-05:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2021-11-17T16:11:26.991-05:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:7}] to localhost:27017
2021-11-17T16:11:26.995-05:00 INFO  [connection] Opened connection [connectionId{localValue:9, serverValue:6}] to localhost:27017
2021-11-17T16:11:26.994-05:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:5}] to localhost:27017
2021-11-17T16:11:27.005-05:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:8}] to localhost:27017
2021-11-17T16:11:27.008-05:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:9}] to localhost:27017
2021-11-17T16:11:27.024-05:00 INFO  [connection] Opened connection [connectionId{localValue:10, serverValue:10}] to localhost:27017
2021-11-17T16:11:27.088-05:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2021-11-17T16:11:27.097-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2021-11-17T16:11:27.101-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2021-11-17T16:11:27.124-05:00 INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2021-11-17T16:11:27.129-05:00 INFO  [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2021-11-17T16:11:27.130-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2021-11-17T16:11:27.131-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2021-11-17T16:11:27.131-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2021-11-17T16:11:27.147-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2021-11-17T16:11:27.164-05:00 INFO  [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2021-11-17T16:11:27.166-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2021-11-17T16:11:27.225-05:00 INFO  [LookupTableService] Data Adapter geoip/614e28029f8bf82a3736d378 [@544b3711] STARTING
2021-11-17T16:11:27.387-05:00 INFO  [LookupTableService] Data Adapter geoip/614e28029f8bf82a3736d378 [@544b3711] RUNNING
2021-11-17T16:11:27.388-05:00 INFO  [LookupDataAdapterRefreshService] Adding job for <geoip/614e28029f8bf82a3736d378/@544b3711> [interval=60000ms]
2021-11-17T16:11:27.831-05:00 INFO  [LookupTableService] Cache geoip/614e28469f8bf82a3736d3c2 [@2c50d48d] STARTING
2021-11-17T16:11:27.843-05:00 INFO  [LookupTableService] Cache geoip/614e28469f8bf82a3736d3c2 [@2c50d48d] RUNNING
2021-11-17T16:11:27.941-05:00 INFO  [LookupTableService] Starting lookup table geoip/614e288a9f8bf82a3736d40e [@5bdaef24] using cache geoip/614e28469f8bf82a3736d3c2 [@2c50d48d], data adapter geoip/614e28029f8bf82a3736d378 [@544b3711]
2021-11-17T16:11:28.779-05:00 INFO  [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2021-11-17T16:11:35.602-05:00 INFO  [NetworkListener] Started listener bound to [172.16.2.15:9000]
2021-11-17T16:11:35.606-05:00 INFO  [HttpServer] [HttpServer] Started.
2021-11-17T16:11:35.606-05:00 INFO  [JerseyService] Started REST API at <172.16.2.15:9000>
2021-11-17T16:11:35.607-05:00 INFO  [ServiceManagerListener] Services are healthy
2021-11-17T16:11:35.609-05:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2021-11-17T16:11:35.615-05:00 INFO  [ServerBootstrap] Services started, startup times in ms: {UrlWhitelistService [RUNNING]=68, LocalKafkaMessageQueueWriter [RUNNING]=71, GracefulShutdownService [RUNNING]=76, InputSetupService [RUNNING]=80, FailureHandlingService [RUNNING]=81, LocalKafkaMessageQueueReader [RUNNING]=99, LocalKafkaJournal [RUNNING]=119, JobSchedulerService [RUNNING]=776, PrometheusExporter [RUNNING]=776, EtagService [RUNNING]=776, OutputSetupService [RUNNING]=777, BufferSynchronizerService [RUNNING]=798, UserSessionTerminationService [RUNNING]=829, ConfigurationEtagService [RUNNING]=891, MongoDBProcessingStatusRecorderService [RUNNING]=945, PeriodicalsService [RUNNING]=1071, StreamCacheService [RUNNING]=1142, LookupTableService [RUNNING]=1785, JerseyService [RUNNING]=9457}
2021-11-17T16:11:35.732-05:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/614b86e49f8bf82a3733f849] is now STARTING
2021-11-17T16:11:35.735-05:00 INFO  [ServerBootstrap] Graylog server up and running.
2021-11-17T16:11:36.594-05:00 WARN  [Bootstrap] Unknown channel option 'io.netty.channel.unix.UnixChannelOption#SO_REUSEPORT' for channel '[id: 0x74bb697a]'
2021-11-17T16:11:36.745-05:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/614b86e49f8bf82a3733f849] is now RUNNING
2021-11-17T16:11:36.747-05:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FortiGate, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=0646dbed-0a28-49e5-bf71-00e9e67fcfd9} (channel [id: 0x74bb697a, L:/0:0:0:0:0:0:0:0:1514]) should be >= 262144 but is 212992.
2021-11-17T16:11:44.048-05:00 WARN  [LookupTableService] Lookup table <geoip-lookup> does not exist
2021-11-17T16:12:04.924-05:00 INFO  [connection] Opened connection [connectionId{localValue:11, serverValue:11}] to localhost:27017
2021-11-17T16:12:05.014-05:00 INFO  [connection] Opened connection [connectionId{localValue:12, serverValue:12}] to localhost:27017
10

So Mongo is not throwing errors now but Elastic is

2021-11-17T16:11:07.769-05:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2021-11-17T16:11:12.786-05:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).

-and-

2021-11-17T16:11:28.779-05:00 INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.

are you pointing to the right elasticsearch server in your graylog conf file? You can post it here (obfuscated of course) this grabs relevant info:

cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"

11

Adding on to @tmacgbay suggested, could you show use your Elasticsearch file also?

cat /etc/elasticsearch/elasticsearch.yml | egrep -v "^\s*(#|$)"

EDIT: When you stated you rolled back your GL server to a working order then posted your ES status before Graylog restarted. I see some errors/warnings it looks familiar but I’m not exactly sure what it could be.

For example:

Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at org.elasticsearch.action.search.AbstractSearchAsyncAction.raisePhaseFailure(AbstractSearchAsyncActi
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:100)
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39)
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(Thre
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
Nov 16 15:36:11 graylog systemd-entrypoint[1099]:         at java.base/java.lang.Thread.run(Thread.java:832)

Just to check roll back you GL server to working order and execute these commands.
This will check to make insure ES is working correctly making sure you don’t have any problems before restarting GL.

ES Health Check.

curl -XGET http://localhost:9200/_cluster/health?pretty=true

ES cluster Info. This will explain why.

curl -XGET http://localhost:9200/_cluster/allocation/explain?pretty

Check indices

curl -XGET http://localhost:9200/_cat/indices?pretty

They may help us troubleshoot this issue. Could you show your Elasticsearch Logs. Should be located in here.

/var/log/elasticsearch

Thanks

1 Like
12

Thanks for the command line help. That’s always appreciated!

The config files and the ES Health info is exactly the same for the working Graylog server and after the reboot when it’s broken.

Graylog Config:
The actual address of “http_bind_address =” is the Graylog server.

ldog@graylog:~$ cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxx
root_password_sha2 = xxx
root_timezone = America/New_York
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 192.168.1.1:9000
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

Elasticsearch Config:

ldog@graylog:~$ sudo cat /etc/elasticsearch/elasticsearch.yml | egrep -v "^\s*(#|$)"
path.data: /mnt/sdb/data
path.logs: /mnt/sdb/logs
cluster.name: graylog
action.auto_create_index: false

ES Health Check:

ldog@graylog:~$ curl -XGET http://localhost:9200/_cluster/health?pretty=true
{
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 76,
  "active_shards" : 76,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

ES cluster Info:

ldog@graylog:~$ curl -XGET http://localhost:9200/_cluster/allocation/explain?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "unable to find any unassigned shards to explain [ClusterAllocationExplainRequest[useAnyUnassignedShard=true,includeYesDecisions?=false]"
  },
  "status" : 400
}

Indices:

ldog@graylog:~$ curl -XGET http://localhost:9200/_cat/indices?pretty
green open gl-events_1        bCFcSZfMQ-eKEeHKmV69cg 4 0    31961 0  4.8mb  4.8mb
green open gl-events_0        eWftw5n4TXanfKKQEGW98Q 4 0   262366 0 44.2mb 44.2mb
green open gl-events_2        jlRqozEoRtqYBD8JFr8xbA 4 0     6176 0  1.2mb  1.2mb
green open graylog_9          FlmpjkuCSVin_e5VdSrheg 4 0 20042538 0 13.2gb 13.2gb
green open graylog_8          G1NCAGN1S36BttzAdo_4kw 4 0 20000486 0 12.9gb 12.9gb
green open graylog_7          5VSdSddgRkmMf-_A1ApXhQ 4 0 20001081 0 12.7gb 12.7gb
green open graylog_6          rmzKss_qQMyWyDRl87v59g 4 0 20006018 0 12.8gb 12.8gb
green open gl-system-events_2 Ja_1Ja92S6SRbkAW-tDTXw 4 0        0 0   832b   832b
green open graylog_1          15M2yv1PQ52nyowUgT_lSw 4 0 20000554 0 12.4gb 12.4gb
green open graylog_0          NNO1qRxZSQ-STJMYsuBlxg 4 0    25657 0 20.6mb 20.6mb
green open gl-system-events_0 mHqE73gbQV2q8wkV0k2bYw 4 0        0 0   832b   832b
green open gl-system-events_1 ziPQ0e3xReyVFJfUtYl7og 4 0        0 0   832b   832b
green open graylog_5          X_C4mUmISIWLdrl1-iSmPA 4 0 20000577 0 12.8gb 12.8gb
green open graylog_10         4PjLgL9IQXCdBAWWETJjZg 4 0 20003974 0 12.7gb 12.7gb
green open graylog_4          bMpOuO67QCSzfKhA4olF2w 4 0 20000901 0 11.5gb 11.5gb
green open graylog_3          FaL1aEYvRsSLPgpL1yykoQ 4 0 20000740 0 12.8gb 12.8gb
green open graylog_12         np1K3ZpDTqCeiMdvwszc4Q 4 0 17140479 0   12gb   12gb
green open graylog_2          P_5FJ8xZTnODyA13AivRag 4 0 20023151 0 12.3gb 12.3gb
green open graylog_11         QeqRAs0ZTjuP2lTncYKIvg 4 0 20030312 0 12.8gb 12.8gb

I’ll post Elasticsearch logs in a moment.

13

Your Graylog server.conf file looks to me missing the pointer to elasticsearch_hosts which by default points locally. It may be commented out, make sure it isn’t. Also make sure you have localhost pointing to 127.0.0.1 in your host file - it’s usually there and mongo uses localhost later in your config… still worth checking.
Here is the default:
elasticsearch_hosts = https://127.0.0.1:9200

Here is an obfuscated version of my server.conf for posterity - and you can compare for other settings you may want or be missing:

tmacgbay@BHS-44:~$ cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = top secret 
root_username = tmacgbadmin
root_password_sha2 = Sha-when-monkeys
root_email = "GSmith@graylog.org"
root_timezone = America/New_York
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 10.11.12.101:9000
http_enable_cors = true
elasticsearch_hosts = http://eadmin:itselectric@BHES-3:9200
elasticsearch_version = 7
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://BHS-44/graylog?replicaSet=rs0
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
transport_email_enabled = true
transport_email_hostname = mySMTP
transport_email_use_auth = false
transport_email_subject_prefix = [graylog]
transport_email_from_email = graylog@myco.co.uk
transport_email_use_tls = false
transport_email_use_ssl = false
transport_email_web_interface_url = http://BHS-44:9000
proxied_requests_thread_pool_size = 32
14

Still no good.

ldog@graylog:~$ cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = xxx
root_password_sha2 = xxx
root_timezone = America/New_York
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 192.168.1.1:9000
elasticsearch_hosts = http://127.0.0.1:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

Host file:

ldog@graylog:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 ubuntu

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
15

Here are the log files.
/var/log/elasticsearch/graylog.log
/var/log/elasticsearch/gc.log

Are those the files you are interested in, or is there something else?

Graylog.log

ldog@graylog:~$ sudo cat /var/log/elasticsearch/graylog.log
[2021-09-27T08:08:10,987][INFO ][o.e.c.m.MetadataMappingService] [graylog] [graylog_1/ofymN6GNS6GHT56EF_qcMQ] update_mapping [_doc]
[2021-09-27T16:02:06,705][INFO ][o.e.n.Node               ] [graylog] stopping ...
[2021-09-27T16:02:08,619][INFO ][o.e.n.Node               ] [graylog] stopped
[2021-09-27T16:02:08,620][INFO ][o.e.n.Node               ] [graylog] closing ...
[2021-09-27T16:02:08,680][INFO ][o.e.n.Node               ] [graylog] closed

gc.log

ldog@graylog:~$ sudo cat /var/log/elasticsearch/gc.log
[2021-11-18T17:28:09.600+0000][1203][gc] Using G1
[2021-11-18T17:28:10.525+0000][1203][gc,init] Version: 15.0.1+9 (release)
[2021-11-18T17:28:10.525+0000][1203][gc,init] CPUs: 2 total, 2 available
[2021-11-18T17:28:10.525+0000][1203][gc,init] Memory: 3900M
[2021-11-18T17:28:10.525+0000][1203][gc,init] Large Page Support: Disabled
[2021-11-18T17:28:10.525+0000][1203][gc,init] NUMA Support: Disabled
[2021-11-18T17:28:10.525+0000][1203][gc,init] Compressed Oops: Enabled (32-bit)
[2021-11-18T17:28:10.525+0000][1203][gc,init] Heap Region Size: 1M
[2021-11-18T17:28:10.525+0000][1203][gc,init] Heap Min Capacity: 1G
[2021-11-18T17:28:10.525+0000][1203][gc,init] Heap Initial Capacity: 1G
[2021-11-18T17:28:10.525+0000][1203][gc,init] Heap Max Capacity: 1G
[2021-11-18T17:28:10.525+0000][1203][gc,init] Pre-touch: Enabled
[2021-11-18T17:28:10.525+0000][1203][gc,init] Parallel Workers: 2
[2021-11-18T17:28:10.525+0000][1203][gc,init] Concurrent Workers: 1
[2021-11-18T17:28:10.525+0000][1203][gc,init] Concurrent Refinement Workers: 2
[2021-11-18T17:28:10.525+0000][1203][gc,init] Periodic GC: Disabled
[2021-11-18T17:28:10.526+0000][1203][gc,cds ] Mark closed archive regions in map: [0x00000000fff00000, 0x00000000fff7bff8]
[2021-11-18T17:28:10.526+0000][1203][gc,cds ] Mark open archive regions in map: [0x00000000ffe00000, 0x00000000ffe51ff8]
[2021-11-18T17:28:10.526+0000][1203][gc,metaspace] CDS archive(s) mapped at: [0x0000000800000000-0x0000000800b2a000-0x0000000800b2a000), size 11706368, SharedBaseAddress: 0x0000000800000000, ArchiveRelocationMode: 0.
[2021-11-18T17:28:10.526+0000][1203][gc,metaspace] Compressed class space mapped at: 0x0000000800b2c000-0x0000000840b2c000, size: 1073741824
[2021-11-18T17:28:10.526+0000][1203][gc,metaspace] Narrow klass base: 0x0000000800000000, Narrow klass shift: 3, Narrow klass range: 0x100000000
[2021-11-18T17:28:12.504+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1952368200 ns, Reaching safepoint: 4235700 ns, At safepoint: 27800 ns, Total: 4263500 ns
[2021-11-18T17:28:14.205+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1701653900 ns, Reaching safepoint: 225500 ns, At safepoint: 24400 ns, Total: 249900 ns
[2021-11-18T17:28:16.150+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1944524500 ns, Reaching safepoint: 316100 ns, At safepoint: 4900 ns, Total: 321000 ns
[2021-11-18T17:28:17.173+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1022213500 ns, Reaching safepoint: 556700 ns, At safepoint: 39900 ns, Total: 596600 ns
[2021-11-18T17:28:18.099+0000][1203][gc,start    ] GC(0) Pause Young (Normal) (G1 Evacuation Pause)
[2021-11-18T17:28:18.104+0000][1203][gc,task     ] GC(0) Using 2 workers of 2 for evacuation
[2021-11-18T17:28:18.104+0000][1203][gc,age      ] GC(0) Desired survivor size 3670016 bytes, new threshold 15 (max threshold 15)
[2021-11-18T17:28:18.150+0000][1203][gc,age      ] GC(0) Age table with threshold 15 (max threshold 15)
[2021-11-18T17:28:18.150+0000][1203][gc,age      ] GC(0) - age   1:    6621328 bytes,    6621328 total
[2021-11-18T17:28:18.150+0000][1203][gc,phases   ] GC(0)   Pre Evacuate Collection Set: 0.8ms
[2021-11-18T17:28:18.150+0000][1203][gc,phases   ] GC(0)   Merge Heap Roots: 0.1ms
[2021-11-18T17:28:18.150+0000][1203][gc,phases   ] GC(0)   Evacuate Collection Set: 43.0ms
[2021-11-18T17:28:18.150+0000][1203][gc,phases   ] GC(0)   Post Evacuate Collection Set: 1.2ms
[2021-11-18T17:28:18.150+0000][1203][gc,phases   ] GC(0)   Other: 5.9ms
[2021-11-18T17:28:18.150+0000][1203][gc,heap     ] GC(0) Eden regions: 51->0(44)
[2021-11-18T17:28:18.150+0000][1203][gc,heap     ] GC(0) Survivor regions: 0->7(7)
[2021-11-18T17:28:18.150+0000][1203][gc,heap     ] GC(0) Old regions: 0->0
[2021-11-18T17:28:18.150+0000][1203][gc,heap     ] GC(0) Archive regions: 2->2
[2021-11-18T17:28:18.150+0000][1203][gc,heap     ] GC(0) Humongous regions: 1->1
[2021-11-18T17:28:18.150+0000][1203][gc,metaspace] GC(0) Metaspace: 13735K(14336K)->13735K(14336K) NonClass: 12099K(12544K)->12099K(12544K) Class: 1636K(1792K)->1636K(1792K)
[2021-11-18T17:28:18.150+0000][1203][gc          ] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 52M->8M(1024M) 51.334ms
[2021-11-18T17:28:18.150+0000][1203][gc,cpu      ] GC(0) User=0.05s Sys=0.00s Real=0.05s
[2021-11-18T17:28:18.150+0000][1203][safepoint   ] Safepoint "G1CollectForAllocation", Time since last: 925423000 ns, Reaching safepoint: 136300 ns, At safepoint: 51476100 ns, Total: 51612400 ns
[2021-11-18T17:28:19.152+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1000900600 ns, Reaching safepoint: 1240300 ns, At safepoint: 22300 ns, Total: 1262600 ns
[2021-11-18T17:28:19.887+0000][1203][safepoint   ] Safepoint "ICBufferFull", Time since last: 734074800 ns, Reaching safepoint: 190300 ns, At safepoint: 24800 ns, Total: 215100 ns
[2021-11-18T17:28:19.905+0000][1203][safepoint   ] Safepoint "ICBufferFull", Time since last: 17720700 ns, Reaching safepoint: 161900 ns, At safepoint: 26700 ns, Total: 188600 ns
[2021-11-18T17:28:19.920+0000][1203][safepoint   ] Safepoint "ICBufferFull", Time since last: 14781200 ns, Reaching safepoint: 185800 ns, At safepoint: 25800 ns, Total: 211600 ns
[2021-11-18T17:28:19.931+0000][1203][safepoint   ] Safepoint "ICBufferFull", Time since last: 11782900 ns, Reaching safepoint: 158200 ns, At safepoint: 26100 ns, Total: 184300 ns
[2021-11-18T17:28:19.940+0000][1203][safepoint   ] Safepoint "ICBufferFull", Time since last: 8684300 ns, Reaching safepoint: 159100 ns, At safepoint: 26700 ns, Total: 185800 ns
[2021-11-18T17:28:19.997+0000][1203][safepoint   ] Safepoint "ICBufferFull", Time since last: 56764400 ns, Reaching safepoint: 85300 ns, At safepoint: 25500 ns, Total: 110800 ns
[2021-11-18T17:28:20.012+0000][1203][safepoint   ] Safepoint "ICBufferFull", Time since last: 13589400 ns, Reaching safepoint: 817400 ns, At safepoint: 24900 ns, Total: 842300 ns
[2021-11-18T17:28:20.020+0000][1203][safepoint   ] Safepoint "ICBufferFull", Time since last: 7649700 ns, Reaching safepoint: 172700 ns, At safepoint: 27600 ns, Total: 200300 ns
[2021-11-18T17:28:20.031+0000][1203][safepoint   ] Safepoint "ICBufferFull", Time since last: 11229000 ns, Reaching safepoint: 166500 ns, At safepoint: 26500 ns, Total: 193000 ns
[2021-11-18T17:28:20.089+0000][1203][safepoint   ] Safepoint "ICBufferFull", Time since last: 57560100 ns, Reaching safepoint: 94500 ns, At safepoint: 27100 ns, Total: 121600 ns
[2021-11-18T17:28:20.094+0000][1203][safepoint   ] Safepoint "ICBufferFull", Time since last: 4933300 ns, Reaching safepoint: 167000 ns, At safepoint: 28300 ns, Total: 195300 ns
[2021-11-18T17:28:20.108+0000][1203][safepoint   ] Safepoint "ICBufferFull", Time since last: 13568800 ns, Reaching safepoint: 162600 ns, At safepoint: 27000 ns, Total: 189600 ns
[2021-11-18T17:28:20.123+0000][1203][safepoint   ] Safepoint "ICBufferFull", Time since last: 14525100 ns, Reaching safepoint: 1032200 ns, At safepoint: 7700 ns, Total: 1039900 ns
[2021-11-18T17:28:20.131+0000][1203][safepoint   ] Safepoint "ICBufferFull", Time since last: 7253400 ns, Reaching safepoint: 210000 ns, At safepoint: 10100 ns, Total: 220100 ns
[2021-11-18T17:28:20.390+0000][1203][gc,start    ] GC(1) Pause Young (Normal) (G1 Evacuation Pause)
[2021-11-18T17:28:20.390+0000][1203][gc,task     ] GC(1) Using 2 workers of 2 for evacuation
[2021-11-18T17:28:20.390+0000][1203][gc,age      ] GC(1) Desired survivor size 3670016 bytes, new threshold 1 (max threshold 15)
[2021-11-18T17:28:20.421+0000][1203][gc,age      ] GC(1) Age table with threshold 1 (max threshold 15)
[2021-11-18T17:28:20.424+0000][1203][gc,age      ] GC(1) - age   1:    2772656 bytes,    2772656 total
[2021-11-18T17:28:20.424+0000][1203][gc,phases   ] GC(1)   Pre Evacuate Collection Set: 0.2ms
[2021-11-18T17:28:20.424+0000][1203][gc,phases   ] GC(1)   Merge Heap Roots: 0.1ms
[2021-11-18T17:28:20.424+0000][1203][gc,phases   ] GC(1)   Evacuate Collection Set: 30.1ms
[2021-11-18T17:28:20.424+0000][1203][gc,phases   ] GC(1)   Post Evacuate Collection Set: 0.8ms
[2021-11-18T17:28:20.424+0000][1203][gc,phases   ] GC(1)   Other: 2.6ms
[2021-11-18T17:28:20.424+0000][1203][gc,heap     ] GC(1) Eden regions: 44->0(48)
[2021-11-18T17:28:20.424+0000][1203][gc,heap     ] GC(1) Survivor regions: 7->3(7)
[2021-11-18T17:28:20.424+0000][1203][gc,heap     ] GC(1) Old regions: 0->5
[2021-11-18T17:28:20.424+0000][1203][gc,heap     ] GC(1) Archive regions: 2->2
[2021-11-18T17:28:20.424+0000][1203][gc,heap     ] GC(1) Humongous regions: 2->1
[2021-11-18T17:28:20.424+0000][1203][gc,metaspace] GC(1) Metaspace: 16124K(16768K)->16124K(16768K) NonClass: 14168K(14592K)->14168K(14592K) Class: 1956K(2176K)->1956K(2176K)
[2021-11-18T17:28:20.424+0000][1203][gc          ] GC(1) Pause Young (Normal) (G1 Evacuation Pause) 53M->9M(1024M) 34.022ms
[2021-11-18T17:28:20.424+0000][1203][gc,cpu      ] GC(1) User=0.02s Sys=0.00s Real=0.04s
[2021-11-18T17:28:20.424+0000][1203][safepoint   ] Safepoint "G1CollectForAllocation", Time since last: 259112900 ns, Reaching safepoint: 173500 ns, At safepoint: 34120900 ns, Total: 34294400 ns
[2021-11-18T17:28:21.382+0000][1203][gc,start    ] GC(2) Pause Young (Normal) (G1 Evacuation Pause)
[2021-11-18T17:28:21.382+0000][1203][gc,task     ] GC(2) Using 2 workers of 2 for evacuation
[2021-11-18T17:28:21.382+0000][1203][gc,age      ] GC(2) Desired survivor size 3670016 bytes, new threshold 15 (max threshold 15)
[2021-11-18T17:28:21.388+0000][1203][gc,age      ] GC(2) Age table with threshold 15 (max threshold 15)
[2021-11-18T17:28:21.388+0000][1203][gc,age      ] GC(2) - age   1:     542288 bytes,     542288 total
[2021-11-18T17:28:21.388+0000][1203][gc,age      ] GC(2) - age   2:    1810064 bytes,    2352352 total
[2021-11-18T17:28:21.388+0000][1203][gc,phases   ] GC(2)   Pre Evacuate Collection Set: 0.2ms
[2021-11-18T17:28:21.388+0000][1203][gc,phases   ] GC(2)   Merge Heap Roots: 0.1ms
[2021-11-18T17:28:21.388+0000][1203][gc,phases   ] GC(2)   Evacuate Collection Set: 5.4ms
[2021-11-18T17:28:21.388+0000][1203][gc,phases   ] GC(2)   Post Evacuate Collection Set: 0.5ms
[2021-11-18T17:28:21.388+0000][1203][gc,phases   ] GC(2)   Other: 0.3ms
[2021-11-18T17:28:21.388+0000][1203][gc,heap     ] GC(2) Eden regions: 48->0(59)
[2021-11-18T17:28:21.388+0000][1203][gc,heap     ] GC(2) Survivor regions: 3->3(7)
[2021-11-18T17:28:21.388+0000][1203][gc,heap     ] GC(2) Old regions: 5->5
[2021-11-18T17:28:21.388+0000][1203][gc,heap     ] GC(2) Archive regions: 2->2
[2021-11-18T17:28:21.388+0000][1203][gc,heap     ] GC(2) Humongous regions: 1->1
[2021-11-18T17:28:21.388+0000][1203][gc,metaspace] GC(2) Metaspace: 16353K(16768K)->16353K(16768K) NonClass: 14380K(14592K)->14380K(14592K) Class: 1973K(2176K)->1973K(2176K)
[2021-11-18T17:28:21.388+0000][1203][gc          ] GC(2) Pause Young (Normal) (G1 Evacuation Pause) 57M->9M(1024M) 6.718ms
[2021-11-18T17:28:21.388+0000][1203][gc,cpu      ] GC(2) User=0.00s Sys=0.00s Real=0.01s
[2021-11-18T17:28:21.388+0000][1203][safepoint   ] Safepoint "G1CollectForAllocation", Time since last: 957581185 ns, Reaching safepoint: 137577 ns, At safepoint: 6822490 ns, Total: 6960067 ns
[2021-11-18T17:28:22.129+0000][1203][gc,start    ] GC(3) Pause Young (Normal) (G1 Evacuation Pause)
[2021-11-18T17:28:22.129+0000][1203][gc,task     ] GC(3) Using 2 workers of 2 for evacuation
[2021-11-18T17:28:22.129+0000][1203][gc,age      ] GC(3) Desired survivor size 4194304 bytes, new threshold 15 (max threshold 15)
[2021-11-18T17:28:22.152+0000][1203][gc,age      ] GC(3) Age table with threshold 15 (max threshold 15)
[2021-11-18T17:28:22.153+0000][1203][gc,age      ] GC(3) - age   1:    1875480 bytes,    1875480 total
[2021-11-18T17:28:22.153+0000][1203][gc,age      ] GC(3) - age   2:     439224 bytes,    2314704 total
[2021-11-18T17:28:22.153+0000][1203][gc,age      ] GC(3) - age   3:    1810040 bytes,    4124744 total
[2021-11-18T17:28:22.153+0000][1203][gc,phases   ] GC(3)   Pre Evacuate Collection Set: 0.2ms
[2021-11-18T17:28:22.153+0000][1203][gc,phases   ] GC(3)   Merge Heap Roots: 0.1ms
[2021-11-18T17:28:22.153+0000][1203][gc,phases   ] GC(3)   Evacuate Collection Set: 21.9ms
[2021-11-18T17:28:22.153+0000][1203][gc,phases   ] GC(3)   Post Evacuate Collection Set: 0.8ms
[2021-11-18T17:28:22.153+0000][1203][gc,phases   ] GC(3)   Other: 0.9ms
[2021-11-18T17:28:22.153+0000][1203][gc,heap     ] GC(3) Eden regions: 59->0(257)
[2021-11-18T17:28:22.153+0000][1203][gc,heap     ] GC(3) Survivor regions: 3->5(8)
[2021-11-18T17:28:22.153+0000][1203][gc,heap     ] GC(3) Old regions: 5->5
[2021-11-18T17:28:22.153+0000][1203][gc,heap     ] GC(3) Archive regions: 2->2
[2021-11-18T17:28:22.153+0000][1203][gc,heap     ] GC(3) Humongous regions: 1->1
[2021-11-18T17:28:22.153+0000][1203][gc,metaspace] GC(3) Metaspace: 16525K(17024K)->16525K(17024K) NonClass: 14536K(14848K)->14536K(14848K) Class: 1988K(2176K)->1988K(2176K)
[2021-11-18T17:28:22.153+0000][1203][gc          ] GC(3) Pause Young (Normal) (G1 Evacuation Pause) 68M->10M(1024M) 24.214ms
[2021-11-18T17:28:22.153+0000][1203][gc,cpu      ] GC(3) User=0.01s Sys=0.00s Real=0.02s
[2021-11-18T17:28:22.153+0000][1203][safepoint   ] Safepoint "G1CollectForAllocation", Time since last: 740294610 ns, Reaching safepoint: 139172 ns, At safepoint: 24340278 ns, Total: 24479450 ns
[2021-11-18T17:28:23.194+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1040509444 ns, Reaching safepoint: 231213 ns, At safepoint: 13506 ns, Total: 244719 ns
[2021-11-18T17:28:25.128+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1933297593 ns, Reaching safepoint: 232700 ns, At safepoint: 5402 ns, Total: 238102 ns
[2021-11-18T17:28:26.146+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1000335296 ns, Reaching safepoint: 18561370 ns, At safepoint: 10604 ns, Total: 18571974 ns
[2021-11-18T17:28:26.390+0000][1203][gc,start    ] GC(4) Pause Young (Concurrent Start) (Metadata GC Threshold)
[2021-11-18T17:28:26.390+0000][1203][gc,task     ] GC(4) Using 2 workers of 2 for evacuation
[2021-11-18T17:28:26.390+0000][1203][gc,age      ] GC(4) Desired survivor size 17301504 bytes, new threshold 15 (max threshold 15)
[2021-11-18T17:28:26.412+0000][1203][gc,age      ] GC(4) Age table with threshold 15 (max threshold 15)
[2021-11-18T17:28:26.412+0000][1203][gc,age      ] GC(4) - age   1:    2937784 bytes,    2937784 total
[2021-11-18T17:28:26.412+0000][1203][gc,age      ] GC(4) - age   2:     313384 bytes,    3251168 total
[2021-11-18T17:28:26.412+0000][1203][gc,age      ] GC(4) - age   3:     437056 bytes,    3688224 total
[2021-11-18T17:28:26.412+0000][1203][gc,age      ] GC(4) - age   4:    1793960 bytes,    5482184 total
[2021-11-18T17:28:26.412+0000][1203][gc,phases   ] GC(4)   Pre Evacuate Collection Set: 0.2ms
[2021-11-18T17:28:26.412+0000][1203][gc,phases   ] GC(4)   Merge Heap Roots: 0.1ms
[2021-11-18T17:28:26.412+0000][1203][gc,phases   ] GC(4)   Evacuate Collection Set: 20.2ms
[2021-11-18T17:28:26.412+0000][1203][gc,phases   ] GC(4)   Post Evacuate Collection Set: 0.9ms
[2021-11-18T17:28:26.412+0000][1203][gc,phases   ] GC(4)   Other: 0.4ms
[2021-11-18T17:28:26.412+0000][1203][gc,heap     ] GC(4) Eden regions: 184->0(436)
[2021-11-18T17:28:26.412+0000][1203][gc,heap     ] GC(4) Survivor regions: 5->6(33)
[2021-11-18T17:28:26.412+0000][1203][gc,heap     ] GC(4) Old regions: 5->5
[2021-11-18T17:28:26.412+0000][1203][gc,heap     ] GC(4) Archive regions: 2->2
[2021-11-18T17:28:26.412+0000][1203][gc,heap     ] GC(4) Humongous regions: 1->1
[2021-11-18T17:28:26.412+0000][1203][gc,metaspace] GC(4) Metaspace: 20485K(21296K)->20485K(21296K) NonClass: 18037K(18608K)->18037K(18608K) Class: 2448K(2688K)->2448K(2688K)
[2021-11-18T17:28:26.412+0000][1203][gc          ] GC(4) Pause Young (Concurrent Start) (Metadata GC Threshold) 194M->12M(1024M) 22.042ms
[2021-11-18T17:28:26.412+0000][1203][gc,cpu      ] GC(4) User=0.01s Sys=0.00s Real=0.02s
[2021-11-18T17:28:26.412+0000][1203][gc          ] GC(5) Concurrent Cycle
[2021-11-18T17:28:26.413+0000][1203][gc,marking  ] GC(5) Concurrent Clear Claimed Marks
[2021-11-18T17:28:26.413+0000][1203][gc,marking  ] GC(5) Concurrent Clear Claimed Marks 0.027ms
[2021-11-18T17:28:26.413+0000][1203][gc,marking  ] GC(5) Concurrent Scan Root Regions
[2021-11-18T17:28:26.413+0000][1203][safepoint   ] Safepoint "CollectForMetadataAllocation", Time since last: 243726875 ns, Reaching safepoint: 88436 ns, At safepoint: 22390010 ns, Total: 22478446 ns
[2021-11-18T17:28:26.421+0000][1203][gc,marking  ] GC(5) Concurrent Scan Root Regions 8.502ms
[2021-11-18T17:28:26.421+0000][1203][gc,marking  ] GC(5) Concurrent Mark (16.836s)
[2021-11-18T17:28:26.421+0000][1203][gc,marking  ] GC(5) Concurrent Mark From Roots
[2021-11-18T17:28:26.421+0000][1203][gc,task     ] GC(5) Using 1 workers of 1 for marking
[2021-11-18T17:28:26.446+0000][1203][gc,marking  ] GC(5) Concurrent Mark From Roots 24.549ms
[2021-11-18T17:28:26.446+0000][1203][gc,marking  ] GC(5) Concurrent Preclean
[2021-11-18T17:28:26.446+0000][1203][gc,marking  ] GC(5) Concurrent Preclean 0.112ms
[2021-11-18T17:28:26.446+0000][1203][gc,marking  ] GC(5) Concurrent Mark (16.836s, 16.861s) 24.764ms
[2021-11-18T17:28:26.452+0000][1203][gc,start    ] GC(5) Pause Remark
[2021-11-18T17:28:26.459+0000][1203][gc          ] GC(5) Pause Remark 12M->12M(1024M) 7.176ms
[2021-11-18T17:28:26.459+0000][1203][gc,cpu      ] GC(5) User=0.00s Sys=0.00s Real=0.01s
[2021-11-18T17:28:26.459+0000][1203][safepoint   ] Safepoint "G1Concurrent", Time since last: 38978484 ns, Reaching safepoint: 89336 ns, At safepoint: 7276628 ns, Total: 7365964 ns
[2021-11-18T17:28:26.459+0000][1203][gc,marking  ] GC(5) Concurrent Rebuild Remembered Sets
[2021-11-18T17:28:26.487+0000][1203][gc,marking  ] GC(5) Concurrent Rebuild Remembered Sets 28.131ms
[2021-11-18T17:28:26.488+0000][1203][gc,start    ] GC(5) Pause Cleanup
[2021-11-18T17:28:26.489+0000][1203][gc          ] GC(5) Pause Cleanup 12M->12M(1024M) 0.273ms
[2021-11-18T17:28:26.489+0000][1203][gc,cpu      ] GC(5) User=0.00s Sys=0.00s Real=0.00s
[2021-11-18T17:28:26.489+0000][1203][safepoint   ] Safepoint "G1Concurrent", Time since last: 28615515 ns, Reaching safepoint: 701583 ns, At safepoint: 368048 ns, Total: 1069631 ns
[2021-11-18T17:28:26.489+0000][1203][gc,marking  ] GC(5) Concurrent Cleanup for Next Mark
[2021-11-18T17:28:26.492+0000][1203][gc,marking  ] GC(5) Concurrent Cleanup for Next Mark 2.010ms
[2021-11-18T17:28:26.492+0000][1203][gc          ] GC(5) Concurrent Cycle 79.116ms
[2021-11-18T17:28:28.135+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1642606109 ns, Reaching safepoint: 3517044 ns, At safepoint: 11704 ns, Total: 3528748 ns
[2021-11-18T17:28:29.135+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1000151365 ns, Reaching safepoint: 176959 ns, At safepoint: 6002 ns, Total: 182961 ns
[2021-11-18T17:28:30.135+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1000133230 ns, Reaching safepoint: 188059 ns, At safepoint: 6302 ns, Total: 194361 ns
[2021-11-18T17:28:31.142+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1005658353 ns, Reaching safepoint: 338499 ns, At safepoint: 7402 ns, Total: 345901 ns
[2021-11-18T17:28:32.142+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1000167076 ns, Reaching safepoint: 179849 ns, At safepoint: 20906 ns, Total: 200755 ns
[2021-11-18T17:28:33.145+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1001449443 ns, Reaching safepoint: 2172057 ns, At safepoint: 8502 ns, Total: 2180559 ns
[2021-11-18T17:28:34.146+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1000145011 ns, Reaching safepoint: 380191 ns, At safepoint: 11603 ns, Total: 391794 ns
[2021-11-18T17:28:35.152+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1005384611 ns, Reaching safepoint: 222551 ns, At safepoint: 7701 ns, Total: 230252 ns
[2021-11-18T17:28:36.159+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1000489262 ns, Reaching safepoint: 7299441 ns, At safepoint: 31006 ns, Total: 7330447 ns
[2021-11-18T17:28:37.160+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1000140937 ns, Reaching safepoint: 151230 ns, At safepoint: 16003 ns, Total: 167233 ns
[2021-11-18T17:28:37.289+0000][1203][gc,start    ] GC(6) Pause Young (Concurrent Start) (Metadata GC Threshold)
[2021-11-18T17:28:37.289+0000][1203][gc,task     ] GC(6) Using 2 workers of 2 for evacuation
[2021-11-18T17:28:37.289+0000][1203][gc,age      ] GC(6) Desired survivor size 40370176 bytes, new threshold 15 (max threshold 15)
[2021-11-18T17:28:37.369+0000][1203][gc,age      ] GC(6) Age table with threshold 15 (max threshold 15)
[2021-11-18T17:28:37.369+0000][1203][gc,age      ] GC(6) - age   1:   29031424 bytes,   29031424 total
[2021-11-18T17:28:37.369+0000][1203][gc,age      ] GC(6) - age   2:    2842312 bytes,   31873736 total
[2021-11-18T17:28:37.369+0000][1203][gc,age      ] GC(6) - age   3:     313352 bytes,   32187088 total
[2021-11-18T17:28:37.369+0000][1203][gc,age      ] GC(6) - age   4:     436568 bytes,   32623656 total
[2021-11-18T17:28:37.370+0000][1203][gc,age      ] GC(6) - age   5:    1764736 bytes,   34388392 total
[2021-11-18T17:28:37.370+0000][1203][gc,phases   ] GC(6)   Pre Evacuate Collection Set: 0.3ms
[2021-11-18T17:28:37.370+0000][1203][gc,phases   ] GC(6)   Merge Heap Roots: 0.2ms
[2021-11-18T17:28:37.370+0000][1203][gc,phases   ] GC(6)   Evacuate Collection Set: 77.6ms
[2021-11-18T17:28:37.370+0000][1203][gc,phases   ] GC(6)   Post Evacuate Collection Set: 1.9ms
[2021-11-18T17:28:37.370+0000][1203][gc,phases   ] GC(6)   Other: 0.4ms
[2021-11-18T17:28:37.370+0000][1203][gc,heap     ] GC(6) Eden regions: 233->0(129)
[2021-11-18T17:28:37.370+0000][1203][gc,heap     ] GC(6) Survivor regions: 6->34(77)
[2021-11-18T17:28:37.370+0000][1203][gc,heap     ] GC(6) Old regions: 5->5
[2021-11-18T17:28:37.370+0000][1203][gc,heap     ] GC(6) Archive regions: 2->2
[2021-11-18T17:28:37.370+0000][1203][gc,heap     ] GC(6) Humongous regions: 1->1
[2021-11-18T17:28:37.370+0000][1203][gc,metaspace] GC(6) Metaspace: 34560K(35920K)->34560K(35920K) NonClass: 30164K(31056K)->30164K(31056K) Class: 4396K(4864K)->4396K(4864K)
[2021-11-18T17:28:37.370+0000][1203][gc          ] GC(6) Pause Young (Concurrent Start) (Metadata GC Threshold) 245M->40M(1024M) 80.387ms
[2021-11-18T17:28:37.370+0000][1203][gc,cpu      ] GC(6) User=0.15s Sys=0.00s Real=0.08s
[2021-11-18T17:28:37.370+0000][1203][gc          ] GC(7) Concurrent Cycle
[2021-11-18T17:28:37.370+0000][1203][gc,marking  ] GC(7) Concurrent Clear Claimed Marks
[2021-11-18T17:28:37.370+0000][1203][gc,marking  ] GC(7) Concurrent Clear Claimed Marks 0.064ms
[2021-11-18T17:28:37.370+0000][1203][gc,marking  ] GC(7) Concurrent Scan Root Regions
[2021-11-18T17:28:37.370+0000][1203][safepoint   ] Safepoint "CollectForMetadataAllocation", Time since last: 129389206 ns, Reaching safepoint: 120023 ns, At safepoint: 80718674 ns, Total: 80838697 ns
[2021-11-18T17:28:37.402+0000][1203][safepoint   ] Safepoint "CollectForMetadataAllocation", Time since last: 31998032 ns, Reaching safepoint: 119624 ns, At safepoint: 173034 ns, Total: 292658 ns
[2021-11-18T17:28:37.457+0000][1203][gc,marking  ] GC(7) Concurrent Scan Root Regions 86.800ms
[2021-11-18T17:28:37.457+0000][1203][gc,marking  ] GC(7) Concurrent Mark (27.872s)
[2021-11-18T17:28:37.457+0000][1203][gc,marking  ] GC(7) Concurrent Mark From Roots
[2021-11-18T17:28:37.457+0000][1203][gc,task     ] GC(7) Using 1 workers of 1 for marking
[2021-11-18T17:28:37.476+0000][1203][gc,marking  ] GC(7) Concurrent Mark From Roots 19.211ms
[2021-11-18T17:28:37.476+0000][1203][gc,marking  ] GC(7) Concurrent Preclean
[2021-11-18T17:28:37.476+0000][1203][gc,marking  ] GC(7) Concurrent Preclean 0.077ms
[2021-11-18T17:28:37.476+0000][1203][gc,marking  ] GC(7) Concurrent Mark (27.872s, 27.891s) 19.373ms
[2021-11-18T17:28:37.477+0000][1203][gc,start    ] GC(7) Pause Remark
[2021-11-18T17:28:37.483+0000][1203][gc          ] GC(7) Pause Remark 43M->43M(1024M) 5.864ms
[2021-11-18T17:28:37.483+0000][1203][gc,cpu      ] GC(7) User=0.01s Sys=0.00s Real=0.00s
[2021-11-18T17:28:37.483+0000][1203][safepoint   ] Safepoint "G1Concurrent", Time since last: 73872019 ns, Reaching safepoint: 541807 ns, At safepoint: 5935575 ns, Total: 6477382 ns
[2021-11-18T17:28:37.483+0000][1203][gc,marking  ] GC(7) Concurrent Rebuild Remembered Sets
[2021-11-18T17:28:37.496+0000][1203][gc,marking  ] GC(7) Concurrent Rebuild Remembered Sets 12.940ms
[2021-11-18T17:28:37.500+0000][1203][gc,start    ] GC(7) Pause Cleanup
[2021-11-18T17:28:37.501+0000][1203][gc          ] GC(7) Pause Cleanup 43M->43M(1024M) 0.196ms
[2021-11-18T17:28:37.501+0000][1203][gc,cpu      ] GC(7) User=0.00s Sys=0.00s Real=0.00s
[2021-11-18T17:28:37.501+0000][1203][safepoint   ] Safepoint "G1Concurrent", Time since last: 13197611 ns, Reaching safepoint: 4538398 ns, At safepoint: 263953 ns, Total: 4802351 ns
[2021-11-18T17:28:37.501+0000][1203][gc,marking  ] GC(7) Concurrent Cleanup for Next Mark
[2021-11-18T17:28:37.503+0000][1203][gc,marking  ] GC(7) Concurrent Cleanup for Next Mark 2.226ms
[2021-11-18T17:28:37.503+0000][1203][gc          ] GC(7) Concurrent Cycle 133.194ms
[2021-11-18T17:28:39.432+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1930552070 ns, Reaching safepoint: 1273922 ns, At safepoint: 9101 ns, Total: 1283023 ns
[2021-11-18T17:28:40.457+0000][1203][safepoint   ] Safepoint "Cleanup", Time since last: 1023179618 ns, Reaching safepoint: 1744284 ns, At safepoint: 12302 ns, Total: 1756586 ns
[2021-11-18T17:28:41.023+0000][1203][safepoint   ] Safepoint "ICBufferFull", Time since last: 564930390 ns, Reaching safepoint: 292245 ns, At safepoint: 7701 ns, Total: 299946 ns
[2021-11-18T17:28:41.870+0000][1203][gc,start    ] GC(8) Pause Young (Concurrent Start) (Metadata GC Threshold)
[2021-11-18T17:28:41.870+0000][1203][gc,task     ] GC(8) Using 2 workers of 2 for evacuation
[2021-11-18T17:28:41.870+0000][1203][gc,age      ] GC(8) Desired survivor size 11534336 bytes, new threshold 1 (max threshold 15)
[2021-11-18T17:28:42.098+0000][1203][gc,age      ] GC(8) Age table with threshold 1 (max threshold 15)
[2021-11-18T17:28:42.098+0000][1203][gc,age      ] GC(8) - age   1:    7839504 bytes,    7839504 total
[2021-11-18T17:28:42.098+0000][1203][gc,mmu      ] GC(8) MMU target violated: 201.0ms (200.0ms/201.0ms)
[2021-11-18T17:28:42.098+0000][1203][gc,phases   ] GC(8)   Pre Evacuate Collection Set: 0.4ms
[2021-11-18T17:28:42.098+0000][1203][gc,phases   ] GC(8)   Merge Heap Roots: 0.2ms
[2021-11-18T17:28:42.098+0000][1203][gc,phases   ] GC(8)   Evacuate Collection Set: 223.7ms
[2021-11-18T17:28:42.098+0000][1203][gc,phases   ] GC(8)   Post Evacuate Collection Set: 3.6ms
[2021-11-18T17:28:42.098+0000][1203][gc,phases   ] GC(8)   Other: 0.5ms
[2021-11-18T17:28:42.098+0000][1203][gc,heap     ] GC(8) Eden regions: 122->0(139)
[2021-11-18T17:28:42.098+0000][1203][gc,heap     ] GC(8) Survivor regions: 34->8(22)
[2021-11-18T17:28:42.098+0000][1203][gc,heap     ] GC(8) Old regions: 5->38
[2021-11-18T17:28:42.099+0000][1203][gc,heap     ] GC(8) Archive regions: 2->2
[2021-11-18T17:28:42.099+0000][1203][gc,heap     ] GC(8) Humongous regions: 1->1
[2021-11-18T17:28:42.099+0000][1203][gc,metaspace] GC(8) Metaspace: 59221K(60976K)->59221K(60976K) NonClass: 51550K(52656K)->51550K(52656K) Class: 7670K(8320K)->7670K(8320K)
[2021-11-18T17:28:42.099+0000][1203][gc          ] GC(8) Pause Young (Concurrent Start) (Metadata GC Threshold) 162M->47M(1024M) 228.701ms
[2021-11-18T17:28:42.099+0000][1203][gc,cpu      ] GC(8) User=0.22s Sys=0.00s Real=0.23s
[2021-11-18T17:28:42.099+0000][1203][safepoint   ] Safepoint "CollectForMetadataAllocation", Time since last: 846764049 ns, Reaching safepoint: 462070 ns, At safepoint: 229172120 ns, Total: 229634190 ns
[2021-11-18T17:28:42.099+0000][1203][gc          ] GC(9) Concurrent Cycle
[2021-11-18T17:28:42.099+0000][1203][gc,marking  ] GC(9) Concurrent Clear Claimed Marks
[2021-11-18T17:28:42.099+0000][1203][gc,marking  ] GC(9) Concurrent Clear Claimed Marks 0.094ms
[2021-11-18T17:28:42.099+0000][1203][gc,marking  ] GC(9) Concurrent Scan Root Regions
[2021-11-18T17:28:42.439+0000][1203][safepoint   ] Safepoint "CollectForMetadataAllocation", Time since last: 329712254 ns, Reaching safepoint: 9317436 ns, At safepoint: 561780 ns, Total: 9879216 ns
[2021-11-18T17:28:42.655+0000][1203][safepoint   ] Safepoint "CollectForMetadataAllocation", Time since last: 216177083 ns, Reaching safepoint: 354751 ns, At safepoint: 302443 ns, Total: 657194 ns
[2021-11-18T17:28:42.713+0000][1203][gc,marking  ] GC(9) Concurrent Scan Root Regions 614.148ms
[2021-11-18T17:28:42.714+0000][1203][gc,marking  ] GC(9) Concurrent Mark (33.128s)
[2021-11-18T17:28:42.714+0000][1203][gc,marking  ] GC(9) Concurrent Mark From Roots
[2021-11-18T17:28:42.714+0000][1203][gc,task     ] GC(9) Using 1 workers of 1 for marking
[2021-11-18T17:28:42.915+0000][1203][gc,marking  ] GC(9) Concurrent Mark From Roots 201.943ms
[2021-11-18T17:28:42.916+0000][1203][gc,marking  ] GC(9) Concurrent Preclean
[2021-11-18T17:28:42.916+0000][1203][gc,marking  ] GC(9) Concurrent Preclean 0.122ms
[2021-11-18T17:28:42.916+0000][1203][gc,marking  ] GC(9) Concurrent Mark (33.128s, 33.331s) 202.177ms
[2021-11-18T17:28:42.929+0000][1203][gc,start    ] GC(9) Pause Remark
[2021-11-18T17:28:42.943+0000][1203][gc          ] GC(9) Pause Remark 63M->63M(1024M) 13.919ms
[2021-11-18T17:28:42.943+0000][1203][gc,cpu      ] GC(9) User=0.02s Sys=0.00s Real=0.02s
[2021-11-18T17:28:42.943+0000][1203][safepoint   ] Safepoint "G1Concurrent", Time since last: 260279504 ns, Reaching safepoint: 13565044 ns, At safepoint: 14084419 ns, Total: 27649463 ns
[2021-11-18T17:28:42.948+0000][1203][gc,marking  ] GC(9) Concurrent Rebuild Remembered Sets
[2021-11-18T17:28:43.170+0000][1203][gc,marking  ] GC(9) Concurrent Rebuild Remembered Sets 222.803ms
[2021-11-18T17:28:43.181+0000][1203][gc,start    ] GC(9) Pause Cleanup
[2021-11-18T17:28:43.182+0000][1203][gc          ] GC(9) Pause Cleanup 66M->66M(1024M) 0.389ms
[2021-11-18T17:28:43.182+0000][1203][gc,cpu      ] GC(9) User=0.00s Sys=0.00s Real=0.00s
[2021-11-18T17:28:43.182+0000][1203][safepoint   ] Safepoint "G1Concurrent", Time since last: 231931698 ns, Reaching safepoint: 5949099 ns, At safepoint: 497867 ns, Total: 6446966 ns
16

Hard to figure out what we are missing!! Here is something… your elaticsearch.yml in a previous post says your data is hanging off /mnt (non-default but fine) but when you posted logs from /var/log/elasticsearch they have todays timestamp. Link or misconfiguration or I am missing something?

and

1 Like
17

That’s a good point. Yes, I have a second vhdx on a SSD as a data drive.

I was wondering if it was not being mounted on reboot but it looks like it is.

ldog@graylog:~$ sudo mount | grep sdb
/dev/sdb on /mnt/sdb type ext4 (rw,relatime,data=ordered)

Below are the files in each of the log directories.
/mnt/sdb/logs/
(I did not list all the files. It was more of “graylog-2021-xx-xx-1.json.gz” going back in time.)

ldog@graylog:~$ ls -l /mnt/sdb/logs/
-rw-r--r-- 1 elasticsearch elasticsearch   256 Nov  3 13:01 graylog-2021-11-02-1.json.gz
-rw-r--r-- 1 elasticsearch elasticsearch   138 Nov  3 13:01 graylog-2021-11-02-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch   582 Nov  7 11:51 graylog-2021-11-03-1.json.gz
-rw-r--r-- 1 elasticsearch elasticsearch   429 Nov  7 11:51 graylog-2021-11-03-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch   563 Nov 11 12:41 graylog-2021-11-07-1.json.gz
-rw-r--r-- 1 elasticsearch elasticsearch   413 Nov 11 12:41 graylog-2021-11-07-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch   583 Nov 12 01:01 graylog-2021-11-11-1.json.gz
-rw-r--r-- 1 elasticsearch elasticsearch   433 Nov 12 01:01 graylog-2021-11-11-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch   339 Nov 15 16:50 graylog-2021-11-12-1.json.gz
-rw-r--r-- 1 elasticsearch elasticsearch   222 Nov 15 16:50 graylog-2021-11-12-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch   341 Nov 16 14:41 graylog-2021-11-15-1.json.gz
-rw-r--r-- 1 elasticsearch elasticsearch   220 Nov 16 14:41 graylog-2021-11-15-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch  1874 Nov 17 04:22 graylog-2021-11-16-1.json.gz
-rw-r--r-- 1 elasticsearch elasticsearch  1716 Nov 17 04:22 graylog-2021-11-16-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch  1438 Nov 18 11:19 graylog-2021-11-17-1.json.gz
-rw-r--r-- 1 elasticsearch elasticsearch  1287 Nov 18 11:19 graylog-2021-11-17-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch     0 Sep 21 15:33 graylog_deprecation.json
-rw-r--r-- 1 elasticsearch elasticsearch     0 Sep 21 15:33 graylog_deprecation.log
-rw-r--r-- 1 elasticsearch elasticsearch     0 Sep 21 15:33 graylog_index_indexing_slowlog.json
-rw-r--r-- 1 elasticsearch elasticsearch     0 Sep 21 15:33 graylog_index_indexing_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch     0 Sep 21 15:33 graylog_index_search_slowlog.json
-rw-r--r-- 1 elasticsearch elasticsearch     0 Sep 21 15:33 graylog_index_search_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 10735 Nov 18 14:02 graylog.log
-rw-r--r-- 1 elasticsearch elasticsearch 13889 Nov 18 14:02 graylog_server.json

/var/log/elasticsearch/

ldog@graylog:~$ sudo ls -l /var/log/elasticsearch/
total 2012512
-rw-r--r-- 1 elasticsearch elasticsearch 46370837 Nov 18 14:51 gc.log
-rw-r--r-- 1 elasticsearch elasticsearch 67108904 Oct 19 22:15 gc.log.00
-rw-r--r-- 1 elasticsearch elasticsearch 67108961 Oct 20 12:28 gc.log.01
-rw-r--r-- 1 elasticsearch elasticsearch 67108908 Oct 22 14:00 gc.log.02
-rw-r--r-- 1 elasticsearch elasticsearch 67109019 Oct 24 17:51 gc.log.03
-rw-r--r-- 1 elasticsearch elasticsearch 67108964 Oct 25 17:55 gc.log.04
-rw-r--r-- 1 elasticsearch elasticsearch 67108893 Oct 26 03:18 gc.log.05
-rw-r--r-- 1 elasticsearch elasticsearch 67108866 Oct 26 13:16 gc.log.06
-rw-r--r-- 1 elasticsearch elasticsearch 67108886 Oct 27 02:04 gc.log.07
-rw-r--r-- 1 elasticsearch elasticsearch 67108923 Oct 27 15:24 gc.log.08
-rw-r--r-- 1 elasticsearch elasticsearch 67108873 Oct 28 04:45 gc.log.09
-rw-r--r-- 1 elasticsearch elasticsearch 67109027 Oct 28 17:42 gc.log.10
-rw-r--r-- 1 elasticsearch elasticsearch 67108958 Oct 29 06:41 gc.log.11
-rw-r--r-- 1 elasticsearch elasticsearch 67108887 Oct 30 10:31 gc.log.12
-rw-r--r-- 1 elasticsearch elasticsearch 67108878 Nov  1 14:38 gc.log.13
-rw-r--r-- 1 elasticsearch elasticsearch 67108923 Nov  3 22:41 gc.log.14
-rw-r--r-- 1 elasticsearch elasticsearch 67108907 Nov  5 22:59 gc.log.15
-rw-r--r-- 1 elasticsearch elasticsearch 67108932 Nov  8 09:11 gc.log.16
-rw-r--r-- 1 elasticsearch elasticsearch 67108920 Nov 11 07:53 gc.log.17
-rw-r--r-- 1 elasticsearch elasticsearch 67108876 Nov 16 00:20 gc.log.18
-rw-r--r-- 1 elasticsearch elasticsearch 67108913 Nov 16 18:29 gc.log.19
-rw-r--r-- 1 elasticsearch elasticsearch 67108941 Nov 17 11:39 gc.log.20
-rw-r--r-- 1 elasticsearch elasticsearch   870437 Oct 13 10:06 gc.log.21
-rw-r--r-- 1 elasticsearch elasticsearch     2016 Oct 13 10:08 gc.log.22
-rw-r--r-- 1 elasticsearch elasticsearch 67108903 Oct 14 19:17 gc.log.23
-rw-r--r-- 1 elasticsearch elasticsearch 67108898 Oct 15 10:04 gc.log.24
-rw-r--r-- 1 elasticsearch elasticsearch 67108888 Oct 16 00:28 gc.log.25
-rw-r--r-- 1 elasticsearch elasticsearch 67108884 Oct 16 15:25 gc.log.26
-rw-r--r-- 1 elasticsearch elasticsearch 67108910 Oct 17 05:39 gc.log.27
-rw-r--r-- 1 elasticsearch elasticsearch 67109012 Oct 17 18:05 gc.log.28
-rw-r--r-- 1 elasticsearch elasticsearch 67108864 Oct 18 04:38 gc.log.29
-rw-r--r-- 1 elasticsearch elasticsearch 67108964 Oct 18 17:15 gc.log.30
-rw-r--r-- 1 elasticsearch elasticsearch 67108905 Oct 19 07:28 gc.log.31
-rw-r--r-- 1 elasticsearch elasticsearch     5353 Sep 22 15:03 graylog-2021-09-21-1.json.gz
-rw-r--r-- 1 elasticsearch elasticsearch     5015 Sep 22 15:03 graylog-2021-09-21-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch      361 Sep 25 15:51 graylog-2021-09-22-1.json.gz
-rw-r--r-- 1 elasticsearch elasticsearch      216 Sep 25 15:51 graylog-2021-09-22-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch      258 Sep 26 18:51 graylog-2021-09-25-1.json.gz
-rw-r--r-- 1 elasticsearch elasticsearch      139 Sep 26 18:51 graylog-2021-09-25-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch      577 Sep 27 08:08 graylog-2021-09-26-1.json.gz
-rw-r--r-- 1 elasticsearch elasticsearch      426 Sep 27 08:08 graylog-2021-09-26-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch        0 Sep 21 10:48 graylog_deprecation.json
-rw-r--r-- 1 elasticsearch elasticsearch        0 Sep 21 10:48 graylog_deprecation.log
-rw-r--r-- 1 elasticsearch elasticsearch        0 Sep 21 10:48 graylog_index_indexing_slowlog.json
-rw-r--r-- 1 elasticsearch elasticsearch        0 Sep 21 10:48 graylog_index_indexing_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch        0 Sep 21 10:48 graylog_index_search_slowlog.json
-rw-r--r-- 1 elasticsearch elasticsearch        0 Sep 21 10:48 graylog_index_search_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch      452 Sep 27 16:02 graylog.log
-rw-r--r-- 1 elasticsearch elasticsearch     1392 Sep 27 16:02 graylog_server.json

So it looks like I was giving you the wrong log file!
/mnt/sdb/logs/graylog.log

ldog@graylog:/mnt/sdb/logs$ cat graylog.log
[2021-11-18T11:19:32,778][INFO ][o.e.c.m.MetadataMappingService] [graylog] [graylog_12/np1K3ZpDTqCeiMdvwszc4Q] update_mapping [_doc]
[2021-11-18T11:19:33,067][INFO ][o.e.c.m.MetadataMappingService] [graylog] [graylog_12/np1K3ZpDTqCeiMdvwszc4Q] update_mapping [_doc]
[2021-11-18T11:19:33,224][INFO ][o.e.c.m.MetadataMappingService] [graylog] [graylog_12/np1K3ZpDTqCeiMdvwszc4Q] update_mapping [_doc]
[2021-11-18T11:19:33,234][INFO ][o.e.c.m.MetadataMappingService] [graylog] [graylog_12/np1K3ZpDTqCeiMdvwszc4Q] update_mapping [_doc]
[2021-11-18T11:19:33,501][INFO ][o.e.c.m.MetadataMappingService] [graylog] [graylog_12/np1K3ZpDTqCeiMdvwszc4Q] update_mapping [_doc]
[2021-11-18T11:19:33,515][INFO ][o.e.c.m.MetadataMappingService] [graylog] [graylog_12/np1K3ZpDTqCeiMdvwszc4Q] update_mapping [_doc]
[2021-11-18T11:19:33,754][INFO ][o.e.c.m.MetadataMappingService] [graylog] [graylog_12/np1K3ZpDTqCeiMdvwszc4Q] update_mapping [_doc]
[2021-11-18T11:19:33,984][INFO ][o.e.c.m.MetadataMappingService] [graylog] [graylog_12/np1K3ZpDTqCeiMdvwszc4Q] update_mapping [_doc]
[2021-11-18T11:19:34,246][INFO ][o.e.c.m.MetadataMappingService] [graylog] [graylog_12/np1K3ZpDTqCeiMdvwszc4Q] update_mapping [_doc]
[2021-11-18T11:19:34,466][INFO ][o.e.c.m.MetadataMappingService] [graylog] [graylog_12/np1K3ZpDTqCeiMdvwszc4Q] update_mapping [_doc]
[2021-11-18T11:19:35,793][INFO ][o.e.c.m.MetadataMappingService] [graylog] [graylog_12/np1K3ZpDTqCeiMdvwszc4Q] update_mapping [_doc]
[2021-11-18T11:19:35,949][INFO ][o.e.c.m.MetadataMappingService] [graylog] [graylog_12/np1K3ZpDTqCeiMdvwszc4Q] update_mapping [_doc]
[2021-11-18T11:19:36,215][INFO ][o.e.c.m.MetadataMappingService] [graylog] [graylog_12/np1K3ZpDTqCeiMdvwszc4Q] update_mapping [_doc]
[2021-11-18T14:02:01,336][WARN ][r.suppressed             ] [graylog] path: /_msearch, params: {typed_keys=true}
java.lang.NullPointerException: Cannot invoke "org.elasticsearch.action.search.MultiSearchResponse$Item.isFailure()" because "item" is null
        at org.elasticsearch.action.search.MultiSearchResponse.toXContent(MultiSearchResponse.java:179) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:47) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:42) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:34) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.action.RestCancellableNodeClient$1.onResponse(RestCancellableNodeClient.java:97) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.action.RestCancellableNodeClient$1.onResponse(RestCancellableNodeClient.java:91) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:83) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.TransportMultiSearchAction$1.finish(TransportMultiSearchAction.java:178) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.TransportMultiSearchAction$1.handleResponse(TransportMultiSearchAction.java:164) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.TransportMultiSearchAction$1.onResponse(TransportMultiSearchAction.java:152) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.TransportMultiSearchAction$1.onResponse(TransportMultiSearchAction.java:149) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:83) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.ActionListener$5.onResponse(ActionListener.java:249) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:562) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:119) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:365) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:359) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:230) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:112) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:128) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:47) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:95) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-11-18T14:02:01,228][WARN ][r.suppressed             ] [graylog] path: /_msearch, params: {typed_keys=true}
java.lang.NullPointerException: Cannot invoke "org.elasticsearch.action.search.MultiSearchResponse$Item.isFailure()" because "item" is null
        at org.elasticsearch.action.search.MultiSearchResponse.toXContent(MultiSearchResponse.java:179) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:47) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:42) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:34) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.action.RestCancellableNodeClient$1.onResponse(RestCancellableNodeClient.java:97) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.rest.action.RestCancellableNodeClient$1.onResponse(RestCancellableNodeClient.java:91) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:83) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.TransportMultiSearchAction$1.finish(TransportMultiSearchAction.java:178) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.TransportMultiSearchAction$1.handleResponse(TransportMultiSearchAction.java:164) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.TransportMultiSearchAction$1.onResponse(TransportMultiSearchAction.java:152) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.TransportMultiSearchAction$1.onResponse(TransportMultiSearchAction.java:149) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:83) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.ActionListener$5.onResponse(ActionListener.java:249) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:562) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:119) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:365) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:359) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:230) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$1(FetchSearchPhase.java:112) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:128) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:47) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:95) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-11-18T14:40:34,422][INFO ][o.e.n.Node               ] [graylog] stopping ...
[2021-11-18T14:40:42,687][INFO ][o.e.n.Node               ] [graylog] stopped
[2021-11-18T14:40:42,688][INFO ][o.e.n.Node               ] [graylog] closing ...
[2021-11-18T14:40:43,426][INFO ][o.e.n.Node               ] [graylog] closed
[2021-11-18T14:41:44,917][INFO ][o.e.n.Node               ] [graylog] version[7.10.2], pid[1181], build[oss/deb/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.15.0-162-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-11-18T14:41:44,936][INFO ][o.e.n.Node               ] [graylog] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-11-18T14:41:44,937][INFO ][o.e.n.Node               ] [graylog] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-17683608586835313458, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-11-18T14:41:51,109][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [aggs-matrix-stats]
[2021-11-18T14:41:51,110][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [analysis-common]
[2021-11-18T14:41:51,110][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [geo]
[2021-11-18T14:41:51,114][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [ingest-common]
[2021-11-18T14:41:51,114][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [ingest-geoip]
[2021-11-18T14:41:51,123][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [ingest-user-agent]
[2021-11-18T14:41:51,124][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [kibana]
[2021-11-18T14:41:51,125][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [lang-expression]
[2021-11-18T14:41:51,125][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [lang-mustache]
[2021-11-18T14:41:51,131][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [lang-painless]
[2021-11-18T14:41:51,132][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [mapper-extras]
[2021-11-18T14:41:51,133][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [parent-join]
[2021-11-18T14:41:51,140][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [percolator]
[2021-11-18T14:41:51,146][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [rank-eval]
[2021-11-18T14:41:51,147][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [reindex]
[2021-11-18T14:41:51,148][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [repository-url]
[2021-11-18T14:41:51,148][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [systemd]
[2021-11-18T14:41:51,148][INFO ][o.e.p.PluginsService     ] [graylog] loaded module [transport-netty4]
[2021-11-18T14:41:51,159][INFO ][o.e.p.PluginsService     ] [graylog] no plugins loaded
[2021-11-18T14:41:51,388][INFO ][o.e.e.NodeEnvironment    ] [graylog] using [1] data paths, mounts [[/mnt/sdb (/dev/sdb)]], net usable_space [501gb], net total_space [688gb], types [ext4]
[2021-11-18T14:41:51,394][INFO ][o.e.e.NodeEnvironment    ] [graylog] heap size [1gb], compressed ordinary object pointers [true]
[2021-11-18T14:41:52,747][INFO ][o.e.n.Node               ] [graylog] node name [graylog], node ID [Lfg5ABAgRtKaa-BepiwdMw], cluster name [graylog], roles [master, remote_cluster_client, data, ingest]
[2021-11-18T14:42:05,715][INFO ][o.e.t.NettyAllocator     ] [graylog] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2021-11-18T14:42:05,883][INFO ][o.e.d.DiscoveryModule    ] [graylog] using discovery type [zen] and seed hosts providers [settings]
[2021-11-18T14:42:06,517][WARN ][o.e.g.DanglingIndicesState] [graylog] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-11-18T14:42:06,954][INFO ][o.e.n.Node               ] [graylog] initialized
[2021-11-18T14:42:06,955][INFO ][o.e.n.Node               ] [graylog] starting ...
[2021-11-18T14:42:07,265][INFO ][o.e.t.TransportService   ] [graylog] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2021-11-18T14:42:07,914][WARN ][o.e.b.BootstrapChecks    ] [graylog] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2021-11-18T14:42:07,917][INFO ][o.e.c.c.Coordinator      ] [graylog] cluster UUID [fdHrTb1WTaaK4sJgTBXS9A]
[2021-11-18T14:42:07,931][INFO ][o.e.c.c.ClusterBootstrapService] [graylog] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-11-18T14:42:08,102][INFO ][o.e.c.s.MasterService    ] [graylog] elected-as-master ([1] nodes joined)[{graylog}{Lfg5ABAgRtKaa-BepiwdMw}{NRs14pZRTvaREMfVMWaUUQ}{127.0.0.1}{127.0.0.1:9300}{dimr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 6, version: 478, delta: master node changed {previous [], current [{graylog}{Lfg5ABAgRtKaa-BepiwdMw}{NRs14pZRTvaREMfVMWaUUQ}{127.0.0.1}{127.0.0.1:9300}{dimr}]}
[2021-11-18T14:42:08,251][INFO ][o.e.c.s.ClusterApplierService] [graylog] master node changed {previous [], current [{graylog}{Lfg5ABAgRtKaa-BepiwdMw}{NRs14pZRTvaREMfVMWaUUQ}{127.0.0.1}{127.0.0.1:9300}{dimr}]}, term: 6, version: 478, reason: Publication{term=6, version=478}
[2021-11-18T14:42:08,366][INFO ][o.e.h.AbstractHttpServerTransport] [graylog] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2021-11-18T14:42:08,367][INFO ][o.e.n.Node               ] [graylog] started
[2021-11-18T14:42:09,249][INFO ][o.e.g.GatewayService     ] [graylog] recovered [19] indices into cluster_state
[2021-11-18T14:42:29,415][INFO ][o.e.c.r.a.AllocationService] [graylog] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][1], [graylog_0][2], [graylog_0][0]]]).
18

It is interesting you have data in both log areas… at what point did you change the directories in the elastisearch.yml? Both areas have recent file dates - I haven’t moved elastic from the default file areas before… so no experience there… maybe you missed something in the configuration?

Lets also double check that the elastic data is where you expect it to be - looking in
I am not familiar with the data here but when I used the command:

sudo tree /var/lib/elasticsearch
sudo tree /mnt/sdb/data

my default location (/var/lib/elasticsearch) came back with 3011 directories, 15492 files. If you compare the results between your two - maybe we will find something interesting?

You have logs in two areas - it seems likely Elastic is confused on where it is/where it should be…

1 Like
19

Hello.
Maybe I’m wrong but your ES Config doesn’t look right.
Here is my ES Config in my lab.

cluster.name: graylog
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 127.0.0.1
http.port: 9200
action.auto_create_index: false
discovery.type: single-node

And my GL configuration has the following config, as @tmacgbay showed above.

elasticsearch_hosts = https://127.0.0.1:9200

Before you restart elasticsearch service make sure GL service is stopped and restart elasticsearch . When ES is fully started, then start GL service and TAIL your Graylog log file.

EDIT:

This warning can be resolved by adding this to your ES config.

discovery.type: single-node

For this warning please see the link below.

1 Like
20

When I first installed Graylog I specified the alternate file location. However, Graylog did not work. I don’t remember what the problem was. So I blew that VM away and started again. The second time I used that default configurations and let it run for a few days. Then I changed the directories.

Here is the result of the tree command.

sudo tree /mnt/sdb/data
364 directories, 2329 files

sudo tree /var/lib/elasticsearch
76 directories, 604 files

The directory structure of both looks the same.

@gsmith I made those configuration changes. Here is the Graylog log file after restarting the services in the order you said.

tail -f /var/log/graylog-server/server.log

2021-11-19T10:08:39.558-05:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2021-11-19T10:08:41.030-05:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 4.2.0 [org.graylog.aws.AWSPlugin]
2021-11-19T10:08:41.033-05:00 INFO  [CmdLineTool] Loaded plugin: Integrations 4.1.5 [org.graylog.integrations.IntegrationsPlugin]
2021-11-19T10:08:41.035-05:00 INFO  [CmdLineTool] Loaded plugin: Collector 4.2.0 [org.graylog.plugins.collector.CollectorPlugin]
2021-11-19T10:08:41.041-05:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.2.0 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-11-19T10:08:41.042-05:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.2.0+5adccc3 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2021-11-19T10:08:41.042-05:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.2.0+5adccc3 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2021-11-19T10:08:41.082-05:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-11-19T10:08:41.394-05:00 INFO  [Version] HV000001: Hibernate Validator null
2021-11-19T10:08:46.572-05:00 INFO  [InputBufferImpl] Message journal is enabled.
2021-11-19T10:08:46.616-05:00 INFO  [NodeId] Node ID: 0646dbed-0a28-49e5-bf71-00e9e67fcfd9
2021-11-19T10:08:46.935-05:00 INFO  [LogManager] Loading logs.
2021-11-19T10:08:47.043-05:00 INFO  [LogManager] Logs loading complete.
2021-11-19T10:08:47.232-05:00 INFO  [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-11-19T10:08:47.305-05:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2021-11-19T10:08:47.403-05:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-11-19T10:08:47.454-05:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:12}] to localhost:27017
2021-11-19T10:08:47.470-05:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 27]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=13364303}
2021-11-19T10:08:47.501-05:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:13}] to localhost:27017
2021-11-19T10:08:47.916-05:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2021-11-19T10:08:49.274-05:00 INFO  [ElasticsearchVersionProvider] Elasticsearch cluster is running v7.10.2
2021-11-19T10:08:50.651-05:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-11-19T10:08:51.303-05:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-11-19T10:08:51.389-05:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:14}] to localhost:27017
2021-11-19T10:08:52.891-05:00 INFO  [ServerBootstrap] Graylog server 4.2.0+5adccc3 starting up
2021-11-19T10:08:52.892-05:00 INFO  [ServerBootstrap] JRE: Private Build 1.8.0_292 on Linux 4.15.0-162-generic
2021-11-19T10:08:52.893-05:00 INFO  [ServerBootstrap] Deployment: deb
2021-11-19T10:08:52.893-05:00 INFO  [ServerBootstrap] OS: Ubuntu 18.04.6 LTS (bionic)
2021-11-19T10:08:52.893-05:00 INFO  [ServerBootstrap] Arch: amd64
2021-11-19T10:08:52.957-05:00 INFO  [PeriodicalsService] Starting 29 periodicals ...
2021-11-19T10:08:52.958-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2021-11-19T10:08:52.962-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2021-11-19T10:08:52.967-05:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2021-11-19T10:08:52.967-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2021-11-19T10:08:52.968-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2021-11-19T10:08:52.971-05:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2021-11-19T10:08:52.971-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2021-11-19T10:08:52.977-05:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2021-11-19T10:08:53.021-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2021-11-19T10:08:53.029-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2021-11-19T10:08:53.118-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2021-11-19T10:08:53.149-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2021-11-19T10:08:53.175-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2021-11-19T10:08:53.176-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2021-11-19T10:08:53.178-05:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2021-11-19T10:08:53.178-05:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2021-11-19T10:08:53.178-05:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:15}] to localhost:27017
2021-11-19T10:08:53.196-05:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:16}] to localhost:27017
2021-11-19T10:08:53.207-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2021-11-19T10:08:53.213-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2021-11-19T10:08:53.214-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2021-11-19T10:08:53.247-05:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:17}] to localhost:27017
2021-11-19T10:08:53.275-05:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2021-11-19T10:08:53.282-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2021-11-19T10:08:53.285-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2021-11-19T10:08:53.313-05:00 INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2021-11-19T10:08:53.321-05:00 INFO  [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2021-11-19T10:08:53.329-05:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2021-11-19T10:08:53.338-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2021-11-19T10:08:53.349-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2021-11-19T10:08:53.360-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2021-11-19T10:08:53.361-05:00 INFO  [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2021-11-19T10:08:53.362-05:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2021-11-19T10:08:53.371-05:00 INFO  [LookupTableService] Data Adapter geoip/614e28029f8bf82a3736d378 [@66934323] STARTING
2021-11-19T10:08:53.419-05:00 INFO  [LookupTableService] Data Adapter geoip/614e28029f8bf82a3736d378 [@66934323] RUNNING
2021-11-19T10:08:53.419-05:00 INFO  [LookupDataAdapterRefreshService] Adding job for <geoip/614e28029f8bf82a3736d378/@66934323> [interval=60000ms]
2021-11-19T10:08:53.717-05:00 INFO  [LookupTableService] Cache geoip/614e28469f8bf82a3736d3c2 [@73dc6a37] STARTING
2021-11-19T10:08:53.726-05:00 INFO  [LookupTableService] Cache geoip/614e28469f8bf82a3736d3c2 [@73dc6a37] RUNNING
2021-11-19T10:08:53.795-05:00 INFO  [LookupTableService] Starting lookup table geoip/614e288a9f8bf82a3736d40e [@1cc17166] using cache geoip/614e28469f8bf82a3736d3c2 [@73dc6a37], data adapter geoip/614e28029f8bf82a3736d378 [@66934323]
2021-11-19T10:09:00.359-05:00 INFO  [NetworkListener] Started listener bound to [172.16.2.15:9000]
2021-11-19T10:09:00.367-05:00 INFO  [HttpServer] [HttpServer] Started.
2021-11-19T10:09:00.367-05:00 INFO  [JerseyService] Started REST API at <172.16.2.15:9000>
2021-11-19T10:09:00.368-05:00 INFO  [ServiceManagerListener] Services are healthy
2021-11-19T10:09:00.370-05:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2021-11-19T10:09:00.372-05:00 INFO  [ServerBootstrap] Services started, startup times in ms: {FailureHandlingService [RUNNING]=13, InputSetupService [RUNNING]=76, OutputSetupService [RUNNING]=84, UserSessionTerminationService [RUNNING]=84, ConfigurationEtagService [RUNNING]=97, JobSchedulerService [RUNNING]=103, UrlWhitelistService [RUNNING]=105, PrometheusExporter [RUNNING]=110, EtagService [RUNNING]=118, LocalKafkaMessageQueueWriter [RUNNING]=128, LocalKafkaMessageQueueReader [RUNNING]=132, BufferSynchronizerService [RUNNING]=132, GracefulShutdownService [RUNNING]=143, LocalKafkaJournal [RUNNING]=179, MongoDBProcessingStatusRecorderService [RUNNING]=195, StreamCacheService [RUNNING]=264, PeriodicalsService [RUNNING]=426, LookupTableService [RUNNING]=779, JerseyService [RUNNING]=7414}
2021-11-19T10:09:00.384-05:00 INFO  [ServerBootstrap] Graylog server up and running.
2021-11-19T10:09:00.458-05:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/614b86e49f8bf82a3733f849] is now STARTING
2021-11-19T10:09:00.962-05:00 WARN  [Bootstrap] Unknown channel option 'io.netty.channel.unix.UnixChannelOption#SO_REUSEPORT' for channel '[id: 0x1e95df36]'
2021-11-19T10:09:01.038-05:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/614b86e49f8bf82a3733f849] is now RUNNING
2021-11-19T10:09:01.039-05:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FortiGate, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=0646dbed-0a28-49e5-bf71-00e9e67fcfd9} (channel [id: 0x1e95df36, L:/0:0:0:0:0:0:0:0:1514]) should be >= 262144 but is 212992.
2021-11-19T10:09:01.797-05:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:18}] to localhost:27017
2021-11-19T10:09:01.809-05:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:19}] to localhost:27017

I’m still not getting messages…