Graylog Consistently Crashes

Graylog Central (peer support)
1

Please a look at my log. I have tried all I could to stop the crashing but no luck. ES6 reports no error and I would think Mongo too. Only GL4 throws an error. I have done all I can to fix it yet it is not going away. I also learnt that the error may be ignored based on one of the posts in this forum.

2021-05-11 12:13:39,743 INFO o.g.s.b.InputBufferImpl [main] Message journal is enabled.
2021-05-11 12:13:39,770 INFO o.g.p.s.NodeId [main] Node ID: b11cbe64-3832-4865-99bb-04c646799539
2021-05-11 12:13:40,063 INFO o.g.s.k.l.LogManager [main] Loading logs.
2021-05-11 12:13:40,113 INFO o.g.s.k.l.Log [pool-7-thread-1] Completed load of log messagejournal-0 with log end offset 23773631
2021-05-11 12:13:40,133 INFO o.g.s.k.l.LogManager [main] Logs loading complete.
2021-05-11 12:13:40,138 INFO o.g.s.j.KafkaJournal [main] Initialized Kafka based journal at data/journal
2021-05-11 12:13:40,170 INFO o.m.d.cluster [main] Cluster created with settings {hosts=[127.0.4.1:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2021-05-11 12:13:40,225 INFO o.m.d.cluster [main] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-05-11 12:13:40,251 INFO o.m.d.connection [cluster-ClusterId{value='609a74f445aa22409ff2ec81', description='null'}-127.0.4.1:27017] Opened connection [connectionId{localValue:1, serverValue:24490}] to 127.0.4.1:27017
2021-05-11 12:13:40,258 INFO o.m.d.cluster [cluster-ClusterId{value='609a74f445aa22409ff2ec81', description='null'}-127.0.4.1:27017] Monitor thread successfully connected to server with description ServerDescription{address=127.0.4.1:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 21]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=4254263}
2021-05-11 12:13:40,378 INFO o.m.d.connection [main] Opened connection [connectionId{localValue:2, serverValue:24491}] to 127.0.4.1:27017
2021-05-11 12:13:40,636 INFO o.g.s.b.InputBufferImpl [main] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2021-05-11 12:13:40,915 INFO o.g.s.p.ElasticsearchVersionProvider [main] Elasticsearch version set to 6.0.0 - disabling version probe.
2021-05-11 12:13:40,982 INFO i.s.c.AbstractJestClient [main] Setting server pool to a list of 1 servers: [http://127.0.4.1:9200]
2021-05-11 12:13:40,984 INFO i.s.c.JestClientFactory [main] Using multi thread/connection supporting pooling connection manager
2021-05-11 12:13:41,077 INFO i.s.c.JestClientFactory [main] Using custom ObjectMapper instance
2021-05-11 12:13:41,077 INFO i.s.c.JestClientFactory [main] Node Discovery disabled...
2021-05-11 12:13:41,077 INFO i.s.c.JestClientFactory [main] Idle connection reaping disabled...
2021-05-11 12:13:41,653 INFO o.g.s.b.ProcessBuffer [main] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-05-11 12:13:42,160 INFO o.m.d.connection [aws-instance-lookup-refresher-0] Opened connection [connectionId{localValue:3, serverValue:24492}] to 127.0.4.1:27017
2021-05-11 12:13:42,177 INFO o.g.b.OutputBuffer [main] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-05-11 12:13:42,901 INFO o.g.b.ServerBootstrap [main] Graylog server 4.0.6+40b7be5 starting up
2021-05-11 12:13:42,901 INFO o.g.b.ServerBootstrap [main] JRE: OpenJDK BSD Porting Team 1.8.0_292 on FreeBSD 13.0-RELEASE
2021-05-11 12:13:42,901 INFO o.g.b.ServerBootstrap [main] Deployment: unknown
2021-05-11 12:13:57,942 INFO o.g.s.i.PeriodicalsService [PeriodicalsService STARTING] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2021-05-11 12:13:57,942 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2021-05-11 12:13:57,944 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2021-05-11 12:13:57,945 INFO o.g.s.i.PeriodicalsService [PeriodicalsService STARTING] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2021-05-11 12:13:57,944 INFO o.g.p.p.p.LegacyDefaultStreamMigration [periodical-org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration-0] Legacy default stream has no connections, no migration needed.
2021-05-11 12:13:57,982 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2021-05-11 12:13:57,983 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2021-05-11 12:13:57,983 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2021-05-11 12:13:57,983 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2021-05-11 12:14:38,868 INFO o.g.g.h.s.NetworkListener [JerseyService STARTING] Started listener bound to [0.0.0.0:9000]
2021-05-11 12:14:38,870 INFO o.g.g.h.s.HttpServer [JerseyService STARTING] [HttpServer] Started.
2021-05-11 12:14:38,870 INFO o.g.s.i.JerseyService [JerseyService STARTING] Started REST API at <0.0.0.0:9000>
2021-05-11 12:14:38,871 INFO o.g.s.i.ServiceManagerListener [JerseyService STARTING] Services are healthy
2021-05-11 12:14:38,871 INFO o.g.s.i.InputSetupService [eventbus-handler-1] Triggering launching persisted inputs, node transitioned from Uninitialized?[LB:DEAD] to Running?[LB:ALIVE]
2021-05-11 12:14:38,872 INFO o.g.b.ServerBootstrap [main] Services started, startup times in ms: {GracefulShutdownService [RUNNING]=0, UrlWhitelistService [RUNNING]=1, JournalReader [RUNNING]=1, InputSetupService [RUNNING]=2, JobSchedulerService [RUNNING]=2, OutputSetupService [RUNNING]=3, ConfigurationEtagService [RUNNING]=3, BufferSynchronizerService [RUNNING]=4, EtagService [RUNNING]=5, KafkaJournal [RUNNING]=5, MongoDBProcessingStatusRecorderService [RUNNING]=23, LookupTableService [RUNNING]=63, PeriodicalsService [RUNNING]=64, StreamCacheServicentbus-handler-1] Input [Syslog TCP/5e244e8701387f2141b18c4d] is now STARTING
2021-05-11 12:14:38,933 INFO o.g.i.InputStateListener [eventbus-handler-1] Input [Syslog UDP/5fdfcaed60e36d7457ac0f05] is now STARTING
2021-05-11 12:14:38,935 INFO o.g.i.InputStateListener [eventbus-handler-0] Input [Beats/5e22bbe801387f67713f5595] is now STARTING
2021-05-11 12:14:39,021 INFO o.g.i.InputStateListener [eventbus-handler-0] Input [Beats/5e22bbe801387f67713f5595] is now RUNNING
2021-05-11 12:14:39,025 INFO o.g.i.InputStateListener [eventbus-handler-1] Input [Syslog TCP/5e244e8701387f2141b18c4d] is now RUNNING
2021-05-11 12:14:39,031 WARN i.n.b.Bootstrap [inputs-3] Unknown channel option 'io.netty.channel.unix.UnixChannelOption#SO_REUSEPORT' for channel '[id: 0xa227409f]'
2021-05-11 12:14:39,031 WARN i.n.b.Bootstrap [inputs-0] Unknown channel option 'io.netty.channel.unix.UnixChannelOption#SO_REUSEPORT' for channel '[id: 0x76561ff8]'
2021-05-11 12:14:39,044 INFO o.g.i.InputStateListener [eventbus-handler-0] Input [Syslog UDP/5fdfcaed60e36d7457ac0f05] is now RUNNING
2021-05-11 12:14:39,046 INFO o.g.i.InputStateListener [eventbus-handler-1] Input [Syslog UDP/5e22ae8801387fe1b0975a82] is now RUNNING
2021-05-11 12:14:56,072 INFO o.g.c.Server [Thread-2] SIGNAL received. Shutting down.
2021-05-11 12:14:56,075 INFO o.g.s.s.GracefulShutdown [Thread-2] Graceful shutdown initiated.
2021-05-11 12:14:56,076 INFO o.g.s.s.GracefulShutdown [Thread-2] Node status: [Halting?[LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
graylog       85194   0.0  0.0    12876    1868  -  IsJ  08:02    0:00.00 daemon: graylog[85195] (daemon)
2021-04-24 15:21:44,324 WARN o.g.s.p.PluginLoader [main] Plugin directory /plugin does not exist, not loading plugins.
2021-04-24 15:21:44,601 INFO o.g.b.CmdLineTool [main] Running with JVM arguments: -Djava.awt.headless=true -Dapp=graylog -Dlog4j.configurationFile=/usr/local/etc/graylog/log4j2.xml -Xms256m -Xmx1g -XX:-OmitStackTraceInFastThrow -XX:+AggressiveOpts -XX:+CMSClassUnloadingEnabled -XX:+CMSConcurrentMTEnabled -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -XX:+ResizeTLAB -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseFastAccessorMethods -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:NewRatio=1
2021-04-24 15:21:44,887 INFO o.h.v.i.u.Version [main] HV000001: Hibernate Validator null
2021-04-24 15:21:45,111 INFO o.g.s.s.s.SigarService [main] Failed to load SIGAR. Falling back to JMX implementations.
2021-04-24 15:21:47,432 ERROR o.g.b.CmdLineTool [main] Guice error (more detail on log level debug): No implementation for java.util.Map<org.graylog2.plugin.Version, javax.inject.Provider<org.graylog.events.search.MoreSearchAdapter>> was bound.
  Did you mean?
    org.graylog2.plugin.Version annotated with @com.google.inject.name.Named(value=elasticsearch_version) bound  at com.github.joschi.jadconfig.guice.NamedConfigParametersModule.registerParameters(NamedConfigParametersModule.java:80)

    org.graylog.events.search.MoreSearchAdapter bound  at org.graylog2.storage.VersionAwareStorageModule.configure(VersionAwareStorageModule.java:54)

    org.graylog2.plugin.Version annotated with interface org.graylog2.storage.ElasticsearchVersion bound  at org.graylog2.bindings.ElasticsearchModule.configure(ElasticsearchModule.java:28)
...skipping...
al [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] complete, took <0ms>.
2021-05-11 12:24:37,384 INFO o.g.s.i.PeriodicalsService [PeriodicalsService STOPPING] Shutting down periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob].
2021-05-11 12:24:37,384 INFO o.g.s.i.PeriodicalsService [PeriodicalsService STOPPING] Shutdown of periodical [org.graylog.plugins.views.search.db.SearchesCleanUpJob] complete, took <0ms>.
2021-05-11 12:24:37,384 INFO o.g.s.i.PeriodicalsService [PeriodicalsService STOPPING] Shutting down periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp].
2021-05-11 12:24:37,384 INFO o.g.s.i.PeriodicalsService [PeriodicalsService STOPPING] Shutdown of periodical [org.graylog.events.periodicals.EventNotificationStatusCleanUp] complete, took <0ms>.
2021-05-11 12:24:37,384 INFO o.g.s.i.PeriodicalsService [PeriodicalsService STOPPING] Shutting down periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread].
2021-05-11 12:24:37,384 INFO o.g.s.i.PeriodicalsService [PeriodicalsService STOPPING] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2021-05-11 12:24:37,384 INFO o.g.s.s.GracefulShutdown [Thread-2] Goodbye.
2021-05-11 12:24:37,385 INFO o.g.s.i.JerseyService [JerseyService STOPPING] Shutting down HTTP listener at <0.0.0.0:9000>
2021-05-11 12:24:37,390 INFO o.g.l.LookupDataAdapterRefreshService [LookupDataAdapterRefreshService STOPPING] Stopping 0 jobs
2021-05-11 12:24:37,390 INFO o.g.s.k.l.LogManager [KafkaJournal STOPPING] Shutting down.
2021-05-11 12:24:37,447 INFO o.g.g.h.s.NetworkListener [JerseyService STOPPING] Stopped listener bound to [0.0.0.0:9000]
2021-05-11 12:24:37,462 INFO o.g.s.k.l.LogManager [KafkaJournal STOPPING] Shutdown complete.
2021-05-11 12:24:37,465 INFO o.g.s.j.JournalReader [JournalReader] Stopping.
2021-05-11 12:24:38,217 INFO o.g.s.i.ServiceManagerListener [JobSchedulerService] Services are now stopped.
2021-05-11 12:24:40,334 INFO o.g.s.b.InputBufferImpl [main] Message journal is enabled.
2021-05-11 12:24:40,375 INFO o.g.p.s.NodeId [main] Node ID: b11cbe64-3832-4865-99bb-04c646799539
2021-05-11 12:24:40,837 INFO o.g.s.k.l.LogManager [main] Loading logs.
2021-05-11 12:24:40,894 INFO o.g.s.k.l.Log [pool-7-thread-1] Completed load of log messagejournal-0 with log end offset 23778638
2021-05-11 12:24:40,913 INFO o.g.s.k.l.LogManager [main] Logs loading complete.
2021-05-11 12:24:40,918 INFO o.g.s.j.KafkaJournal [main] Initialized Kafka based journal at data/journal
2021-05-11 12:24:40,950 INFO o.m.d.cluster [main] Cluster created with settings {hosts=[127.0.4.1:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2021-05-11 12:24:41,015 INFO o.m.d.cluster [main] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-05-11 12:24:41,036 INFO o.m.d.connection [cluster-ClusterId{value='609a7788ede51e704574a36d', description='null'}-127.0.4.1:27017] Opened connection [connectionId{localValue:1, serverValue:24554}] to 127.0.4.1:27017
2021-05-11 12:24:41,043 INFO o.m.d.cluster [cluster-ClusterId{value='609a7788ede51e704574a36d', description='null'}-127.0.4.1:27017] Monitor thread successfully connected to server with description ServerDescription{address=127.0.4.1:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 21]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=5256961}
2021-05-11 12:24:41,151 INFO o.m.d.connection [main] Opened connection [connectionId{localValue:2, serverValue:24555}] to 127.0.4.1:27017
2021-05-11 12:24:41,489 INFO o.g.s.b.InputBufferImpl [main] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2021-05-11 12:24:41,796 INFO o.g.s.p.ElasticsearchVersionProvider [main] Elasticsearch version set to 6.0.0 - disabling version probe.
2021-05-11 12:24:41,872 INFO i.s.c.AbstractJestClient [main] Setting server pool to a list of 1 servers: [http://127.0.4.1:9200]
2021-05-11 12:24:41,874 INFO i.s.c.JestClientFactory [main] Using multi thread/connection supporting pooling connection manager
2021-05-11 12:24:41,964 INFO i.s.c.JestClientFactory [main] Using custom ObjectMapper instance
2021-05-11 12:24:41,964 INFO i.s.c.JestClientFactory [main] Node Discovery disabled...
2021-05-11 12:24:41,964 INFO i.s.c.JestClientFactory [main] Idle connection reaping disabled...
2021-05-11 12:24:42,614 INFO o.g.s.b.ProcessBuffer [main] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-05-11 12:24:43,241 INFO o.m.d.connection [aws-instance-lookup-refresher-0] Opened connection [connectionId{localValue:3, serverValue:24556}] to 127.0.4.1:27017
2021-05-11 12:24:43,255 INFO o.g.b.OutputBuffer [main] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-05-11 12:24:44,140 INFO o.g.b.ServerBootstrap [main] Graylog server 4.0.6+40b7be5 starting up
2021-05-11 12:24:44,140 INFO o.g.b.ServerBootstrap [main] JRE: OpenJDK BSD Porting Team 1.8.0_292 on FreeBSD 13.0-RELEASE
2021-05-11 12:24:44,141 INFO o.g.b.ServerBootstrap [main] Deployment: unknown
2021-05-11 12:24:44,141 INFO o.g.b.ServerBootstrap [main] OS: FreeBSD
2021-05-11 12:24:44,142 INFO o.g.b.ServerBootstrap [main] Arch: amd64
2021-05-11 12:24:59,178 INFO o.g.s.i.PeriodicalsService [PeriodicalsService STARTING] Starting 30 periodicals ...
2021-05-11 12:24:59,180 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s]
2021-05-11 12:24:59,244 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2021-05-11 12:24:59,246 INFO o.g.s.i.PeriodicalsService [PeriodicalsService STARTING] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2021-05-11 12:24:59,246 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2021-05-11 12:24:59,246 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting 
2021-05-11 12:24:59,249 INFO o.m.d.connection [LookupTableService STARTING] Opened connection [connectionId{localValue:5, serverValue:24558}] to 127.0.4.1:27017
2021-05-11 12:24:59,250 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2021-05-11 12:24:59,251 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2021-05-11 12:24:59,252 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].og2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2021-05-11 12:24:59,271 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting
2021-05-11 12:24:59,275 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2021-05-11 12:24:59,276 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2021-05-11 12:24:59,276 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2021-05-11 12:24:59,276 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2021-05-11 12:24:59,278 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2021-05-11 12:24:59,278 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2021-05-11 12:24:59,278 INFO o.g.p.Periodicals [PeriodicalsService STARTING] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2021-05-11 12:24:59,282 INFO o.m.d.connection [periodical-org.graylog2.periodical.IndexRangesMigrationPeriodical-0] Opened connection [connectionId{localValue:6, serverValue:24559}] to 127.0.4.1:27017




2021-04-24 15:21:44,324 WARN o.g.s.p.PluginLoader [main] Plugin directory /plugin does not exist, not loading plugins.
2021-04-24 15:21:44,601 INFO o.g.b.CmdLineTool [main] Running with JVM arguments: -Djava.awt.headless=true -Dapp=graylog -Dlog4j.configurationFile=/usr/local/etc/graylog/log4j2.xml -Xms256m -Xmx1g -XX:-OmitStackTraceInFastThrow -XX:+AggressiveOpts -XX:+CMSClassUnloadingEnabled -XX:+CMSConcurrentMTEnabled -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -XX:+ResizeTLAB -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseFastAccessorMethods -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:NewRatio=1
2021-04-24 15:21:44,887 INFO o.h.v.i.u.Version [main] HV000001: Hibernate Validator null
2021-04-24 15:21:45,111 INFO o.g.s.s.s.SigarService [main] Failed to load SIGAR. Falling back to JMX implementations.
2021-04-24 15:21:47,432 **ERROR o.g.b.CmdLineTool [main] Guice error (more detail on log level debug): No implementation for java.util.Map<org.graylog2.plugin.Version, javax.inject.Provider<org.graylog.events.search.MoreSearchAdapter>> was bound.**
**  Did you mean?**
    org.graylog2.plugin.Version annotated with @com.google.inject.name.Named(value=elasticsearch_version) bound  at com.github.joschi.jadconfig.guice.NamedConfigParametersModule.registerParameters(NamedConfigParametersModule.java:80)

    org.graylog.events.search.MoreSearchAdapter bound  at org.graylog2.storage.VersionAwareStorageModule.configure(VersionAwareStorageModule.java:54)

    org.graylog2.plugin.Version annotated with interface org.graylog2.storage.ElasticsearchVersion bound  at org.graylog2.bindings.ElasticsearchModule.configure(ElasticsearchModule.java:28)
...skipping...
.1:27017
2021-05-11 12:12:17,969 INFO o.m.d.cluster [cluster-ClusterId{value='609a74a193100d13c97b5c71', description='null'}-127.0.4.1:27017] Monitor thread successfully connected to server with description ServerDescription{address=127.0.4.1:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 21]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=4930269}
2021-05-11 12:12:18,102 INFO o.m.d.connection [main] Opened connection [connectionId{localValue:2, serverValue:24481}] to 127.0.4.1:27017
2021-05-11 12:12:18,381 INFO o.g.s.b.InputBufferImpl [main] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2021-05-11 12:12:18,619 INFO o.g.s.p.ElasticsearchVersionProvider [main] Elasticsearch version set to 6.0.0 - disabling version probe.
2021-05-11 12:12:18,672 INFO i.s.c.AbstractJestClient [main] Setting server pool to a list of 1 servers: [http://127.0.4.1:9200]
2021-05-11 12:12:18,673 INFO i.s.c.JestClientFactory [main] Using multi thread/connection supporting pooling connection manager
2021-05-11 12:12:18,753 INFO i.s.c.JestClientFactory [main] Using custom ObjectMapper instance
2021-05-11 12:12:18,753 INFO i.s.c.JestClientFactory [main] Node Discovery disabled...
2021-05-11 12:12:18,753 INFO i.s.c.JestClientFactory [main] Idle connection reaping disabled...
2021-05-11 12:12:19,354 INFO o.g.s.b.ProcessBuffer [main] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-05-11 12:12:19,944 INFO o.m.d.connection [aws-instance-lookup-refresher-0] Opened connection [connectionId{localValue:3, serverValue:24482}] to 127.0.4.1:27017
2021-05-11 12:12:19,960 INFO o.g.b.OutputBuffer [main] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.

Here is exercpt of Graylog.conf


# Optimize the index down to <= index_optimization_max_num_segments. A higher number may take some load f>
# on heavily used systems with large indices, but it will decrease search performance. The default is 1.
#
# ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure>
#            to your previous settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices'>
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-confi>
#index_optimization_max_num_segments = 1

# The threshold of the garbage collection runs. If GC runs take longer than this threshold, a system noti>
# will be generated to warn the administrator about possible problems with the system. Default is 1 secon>
#gc_warning_threshold = 1s

# Connection timeout for a configured LDAP server (e. g. ActiveDirectory) in milliseconds.
#ldap_connection_timeout = 2000

# Disable the use of SIGAR for collecting system stats
#disable_sigar = false


# Configures the journal write rate threshold for selecting processing status records. Any records that h>
# one minute rate than the configured value might be ignored. (dependent on number of messages in the jou>
# Default: 1
#processing_status_journal_write_rate_threshold = 1

# Default: 10 Seconds
#elasticsearch_connect_timeout = 10s

# Maximum amount of time to wait for reading back a response from an Elasticsearch server.
#
# Default: 60 seconds
#elasticsearch_socket_timeout = 60s

# Maximum idle time for an Elasticsearch connection. If this is exceeded, this connection will
# be tore down.
# 
# Default: inf
#elasticsearch_idle_timeout = -1s

# Maximum number of total connections to Elasticsearch.
#
# Default: 200
#elasticsearch_max_total_connections = 200
#elasticsearch_max_total_connections = 200

# Maximum number of total connections per Elasticsearch route (normally this means per
# elasticsearch server).
#
# Default: 20
#elasticsearch_max_total_connections_per_route = 20

# Maximum number of times Graylog will retry failed requests to Elasticsearch.
#
# Default: 2
#elasticsearch_max_retries = 2

# Enable automatic Elasticsearch node discovery through Nodes Info,
# see https://www.elastic.co/guide/en/elasticsearch/reference/5.4/cluster-nodes-info.html
#
# WARNING: Automatic node discovery does not work if Elasticsearch requires authentication, e. g. with Sh>
#
# Default: false
#elasticsearch_discovery_enabled = true   

#elasticsearch_discovery_enabled = true   

# Filter for including/excluding Elasticsearch nodes in discovery according to their custom attributes,
# see https://www.elastic.co/guide/en/elasticsearch/reference/5.4/cluster.html#cluster-nodes
#
# Default: empty
#elasticsearch_discovery_filter = rack:42

# Frequency of the Elasticsearch node discovery.
#
# Default: 30s
# elasticsearch_discovery_frequency = 30s

# Enable payload compression for Elasticsearch requests.
#
# Default: false
#elasticsearch_compression_enabled = true 

# Graylog will use multiple indices to store documents in. You can configured the strategy it uses to det>
# when to rotate the currently active write index.
# It supports multiple rotation strategies:
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices'>
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-confi>
elasticsearch_max_docs_per_index = 20000000

# (Approximate) maximum size in bytes per Elasticsearch index on disk before a new index is being created>
# no_retention and elasticsearch_max_number_of_indices. Default is 1GB.
# Configure this if you used 'rotation_strategy = size' above.
#            
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set th>
#            to your previous 1.x settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices'>
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-confi>
#elasticsearch_max_size_per_index = 1073741824

# (Approximate) maximum time before a new Elasticsearch index is being created, also see
# no_retention and elasticsearch_max_number_of_indices. Default is 1 day.
# Configure this if you used 'rotation_strategy = time' above.
# Please note that this rotation period does not look at the time 
#elasticsearch_disable_version_check = true

# Disable message retention on this node, i. e. disable Elasticsearch index rotation.
#no_retention = false

# How many indices do you want to keep?
#            
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set th>
#            to your previous 1.x settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices'>
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-confi>
elasticsearch_max_number_of_indices = 20

# Decide what happens with the oldest indices when the maximum number of indices is reached.
# The following strategies are availble:
#   - delete # Deletes the index completely (Default)
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices'>
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-confi>
retention_strategy = delete

# How many Elasticsearch shards and replicas should be used per index? Note that this only applies to new>
# ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure>
#            to your previous settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices'>
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-confi>
elasticsearch_shards = 4
elasticsearch_replicas = 0

# Prefix for all Elasticsearch indices and index aliases managed by Graylog.
#   
# ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure>
#            to your previous settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
# Default: 1m
#elasticsearch_request_timeout = 1m

# Global timeout for index optimization (force merge) requests.
# Default: 1h
#elasticsearch_index_optimization_timeout = 1h

# Maximum number of concurrently running index optimization (force merge) jobs.
# If you are using lots of different index sets, you might want to increase that number.
# Default: 20
#elasticsearch_index_optimization_jobs = 20

# Time interval for index range information cleanups. This setting defines how often stale index range in>
# is being purged from the database.
# Default: 1h
#index_ranges_cleanup_interval = 1h

# Time interval for the job that runs index field type maintenance tasks like cleaning up stale entries. >
# need to run very often.
# Default: 1h
#index_field_type_periodical_interval = 1h

web_enable = true

rest_listen_uri = http://0.0.0.0:9000/
rest_transport_uri =  http://Public_IP:9000/api
web_listen_uri = http://0.0.0.0:9000/

The indices is 9.3G

last pid: 77774;  load averages:  3.50,  3.09,  2.89                              up 6+03:42:38  12:50:22
21 processes:  1 running, 20 sleeping
CPU: 10.0% user,  0.0% nice,  4.9% system,  0.8% interrupt, 84.4% idle
Mem: 2742M Active, 18G Inact, 4012M Laundry, 27G Wired, 10G Free
ARC: 16G Total, 12G MFU, 2258M MRU, 24M Anon, 233M Header, 2183M Other
     12G Compressed, 19G Uncompressed, 1.55:1 Ratio
Swap: 74G Total, 199M Used, 74G Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
77631 graylog      61  52    0  3079M   454M uwait    6   0:34   4.18% java
50845 elasticsea  101  52    0    18G  7253M uwait    5  53:25   0.34% java
18027 mongodb      31  52    0  1092M   180M uwait    4  58:53   0.19% mongod
77740 root          1  20    0    14M  3060K CPU3     3   0:00   0.03% top
83400 www           1  20    0    22M  6328K accept   4   0:03   0.01% httpd
77131 root          1  20    0    20M  6564K select   7   0:00   0.00% curl

Swap capacity is high because it is a build system. I wrote a script that restarts it as soon as there are more than one instance running but no luck.

Apologies for the poor formatting; I could not find the right tags to use.

2

Why do you have your min and max heap set to different values? They should be the same (e.g., -Xms1g-Xmx1g). That’d be one thing to try and see if that addresses the crashes. I’d start there.

3

I played with them hoping they would resolve it; but no luck. Those are the default values after installing from ports. I changed to ‘-Xms512m -Xmx2g’ in /usr/local/etc/rc.d/graylog but returned to the default values when there was no change.

4

What sort of volume are you ingesting? If you’re taking the defaults (e.g., -Xms1g -Xmx1g) but ingesting a large amount of logs (e.g., north of 300-400 events/s), then you’ll need to increase the defaults to -Xms2g -Xmx2g. Also, what version are you running? I’m seeing some values that are no longer used in Graylog (starting with rest_). If you’re running an old version, I’d consider updating.

5

ES=6
GL=4

The rest_* is now replaced with htt_, I think. I shall get rid of it soon since I already have http_. The problem led me to try several strange things.

I’m not ingesting such volume. An average of 70in/20out in the GUI. Would that be the events?

screenshot-graylog_9000-2021.05.11-22_33_11
screenshot-graylog_9000-2021.05.11-22_33_111301×195 23 KB

6

Graylog-4.0.6
Elasticsearch-6.8.15
Mongo-4.0.21

7

Are you running all of the components on the sam instance? If so, what are the specs on your instance?

8
  • Number of CPUs8 (2.4GHz)
  • Memory63.8 GB
  • Swap74 GB

Graylog runs in a jail.

9

Pardon me, YES, they all run in one instance same machine.

10

I’m taking this as a bug. Hopefully, it will be fixed at next release.

11

I’m still looking forward to suggestions from the community.

12

Hello,

I noticed something was not right in your previous post above.
Those settings you showed are for an earlier version of Graylog Not for Graylog Version 4 as shown below. I believe you may have some configuration issues.

This is Graylog 4.0 server.config file shown below.

https://docs.graylog.org/en/4.0/pages/configuration/server.conf.html

If this is correct you may need to install or configure the correct server.config file.
Hope that helps

13

I knew these were old.
I have indeed gotten rid of these lines and matched the config with the 4.0 but no luck yet.

14

@Lamia

Perhaps could we see all you configurations?
Maybe something like this for your Graylog config file.

grep -v "^#\|^$" server.conf | sed -e "s/#.*$//g

15

Apologies for delayed response. Please see below. The posted command was modified because it was throwing an error (Illegal variable name).

grep -Ev ‘^$|^#’ graylog.conf | sed ‘s;//;/;g’

output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = data/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb:/localhost/graylog
mongodb_uri = mongodb:/user:pwd@127.0.4.1:27017/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

16

I reckon the command might not be doing what you intended; here is now the whole conf.


###########################
# GRAYLOG CONFIGURATION FILE
############################
#
# This is the Graylog configuration file. The file has to use ISO 8859-1/Latin-1 character encoding.
# Characters that cannot be directly represented in this encoding can be written using Unicode escapes
# as defined in https://docs.oracle.com/javase/specs/jls/se8/html/jls-3.html#jls-3.3, using the \u prefix.
# For example, \u002c.
#
# * Entries are generally expected to be a single line of the form, one of the following:
#
#
# propertyName=propertyValue
# propertyName:propertyValue
#
# * White space that appears between the property name and property value is ignored,
#   so the following are equivalent:
# 
# name=Stephen
# name = Stephen
#
# * White space at the beginning of the line is also ignored.
#
# * Lines that start with the comment characters ! or # are ignored. Blank lines are also ignored.
#
# * The property value is generally terminated by the end of the line. White space following the
#   property value is not ignored, and is treated as part of the property value.
#
# * A property value can span several lines if each line is terminated by a backslash (‘\’
#         Detroit,\
#         Chicago,\
#         Los Angeles
#
#   This is equivalent to targetCities=Detroit,Chicago,Los Angeles (white space at the beginning of lines is ignored).
# 
# * The characters newline, carriage return, and tab can be inserted with characters \n, \r, and \t, respectively.
# 
# * The backslash character must be escaped as a double backslash. For example:
# 
# path=c:\\docs\\doc1
#

# If you are running more than one instances of Graylog server you have to select one of these
# instances as master. The master will perform some periodical tasks that non-masters won't perform.
is_master = true

# The auto-generated node ID will be stored in this file and read after restarts. It
# The time zone setting of the root user. See http://www.joda.org/joda-time/timezones.html for a list of valid time zones.
# Default is UTC
#root_timezone = UTC

# Set the bin directory here (relative or absolute)
# This directory contains binaries that are used by the Graylog server.
# Default: bin
bin_dir = bin

# Set the data directory here (relative or absolute)
# This directory is used to store Graylog server state.
# Default: data
data_dir = data

# Set plugin directory here (relative or absolute)
# See github.com/KongZ/charts/issues/20
plugin_dir = /usr/local/share/graylog/plugin

###############
# HTTP 
#
# If $http_bind_address contains a wildcard IPv4 address (0.0.0.0), the first non-loopback IPv4 address of this machine will be used.
# This configuration setting *must not* contain a wildcard address!
#
# Default: http://$http_bind_address/
#http_publish_uri = http://192.168.40.20:9000/


#### External Graylog URI
#
# The public URI of Graylog which will be used by the Graylog web interface to communicate with the Graylog REST API.
#
# The external Graylog URI usually has to be specified, if Graylog is running behind a reverse proxy or load-balancer
# and it will be used to generate URLs addressing entities in the Graylog REST API (see $http_bind_address).
#
# When using Graylog Collector, this URI will be used to receive heartbeat messages and must be accessible for all collectors.
#
# This setting
#
# This secures the communication with the HTTP interface with TLS to prevent request forgery and eavesdropping.
#
# Default: false
#http_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the HTTP interface.
#http_tls_cert_file = /path/to/graylog.crt


# The PKCS#8 private key file in PEM format to use for securing the HTTP interface.
#http_tls_key_file = /path/to/graylog.key


# The password to unlock the private key used for securing the HTT
# be tore down.
#
# Default: inf
#elasticsearch_idle_timeout = -1s

# Maximum number of total connections to Elasticsearch.
#
# Default: 200
#elasticsearch_max_total_connections = 200

# Maximum number of total connections per Elasticsearch route (normally this means per
# elasticsearch server).
#
# Default: 20
#elasticsearch_max_total_connections_per_route = 20

# Maximum number of times Graylog will retry failed requests to Elasticsearch.
#
# Default: 2
#elasticsearch_max_retries = 2

# Enable 
e strategy it uses to determine
# when to rotate the currently active write index.
# It supports multiple rotation strategies:
#   - "count" of messages per index, use elasticsearch_max_docs_per_index below to configure
#   - "size" per index, use elasticsearch_max_size_per_index below to configure
# valid values are "count", "size" and "time", default is "count"
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
rotation_strategy = count

# (Approximate) maximum number of documents in an Elasticsearch index before a new index
# is being created, also see no_retention and elasticse
# Configure this if you used 'rotation_strategy = size' above.
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
#elasticsearch_max_size_per_index = 1073741824

# (Approximate) maximum time before a new Elasticsearch index is being created, also see
# no_retention and elasticsearch_max_number_of_indices. Default is 1 day.
# Configure this if you used 'rotation_strategy = time' above.
# Please note that this rotation period does not look at the time specified in the received messages, but is
# using the real clock value to decide when to rotate the index!
# Specify the time using a duration and a suffix indicating which unit you want:
# 
# Disable message retention on this node, i. e. disable Elasticsearch index rotation.
#no_retention = false

# How many indices do you want to keep?
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
elasticsearch_max_number_of_indices = 20

# Decide what happens with the oldest indices when the maximum number of indices is reached.
# The following strategies are availble:
#   - delete # Deletes the index completely (Default)
#   - close # Closes the index and hide
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
elasticsearch_shards = 4
elasticsearch_replicas = 0

# Prefix for all Elasticsearch indices and index aliases managed by Graylog.
#
# ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
#            to your previous settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
elasticsearch_index_prefix = graylog

# Name of the Elastics

# Do you want to allow searches to be highlighted? Depending on the size of your messages this can be memory hungry and
# should only be enabled after making sure your Elasticsearch cluster has enough memory.
allow_highlighting = false

# Analyzer (tokenizer) to use for message and full_message field. The "standard" filter usually is a good idea.
# All supported analyzers are: standard, simple, whitespace, stop, keyword, pattern, language, snowball, custom
# Elasticsearch documentation: https://www.elastic.co/guide/en/elasticsearch/reference/2.3/analysis.html
# Note that this setting only takes effect on newly created indices.
#
# ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
#            to your previous settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.


# Time interval for the job that runs index field type maintenance tasks like cleaning up stale entries. This doesn't
# need to run very often.
# Default: 1h
#index_field_type_periodical_interval = 1h

# Batch size for the Elasticsearch output. This is the maximum (!) number of messages the Elasticsearch output
# module will get at once and write to Elasticsearch in a batch call. If the configured batch size has not been
# reached within output_flush_interval seconds, everything that is available will be flushed at once. Remember
# that every outputbuffer processor manages its own batch and performs its own batch write calls.
# ("outputbuffer_processors" variable)
output_batch_size = 500

# Flush interval (in seconds) for the Elasticsearch output. This is the maximum amount of time between two
# batches of messages written to Elasticsearch. It is only effective at all if your minimum number of messages
# for this time period is less than output_batch_size * outputbuffer_pro
# Default: 3
#outputbuffer_processor_threads_core_pool_size = 3

# The maximum number of threads to allow in the pool
# Default: 30
#outputbuffer_processor_threads_max_pool_size = 30

# UDP receive buffer size for all message inputs (e. g. SyslogUDPInput).
#udp_recvbuffer_sizes = 1048576

# Wait strategy describing how buffer processors wait on a cursor sequence. (default: sleeping)
# Possible types:
#  - yielding
#     Compromise between performance and CPU usage.
#  - sleeping
#     Compromise between performance and CPU usage. Latency spikes can occur after quiet periods.
#  - blocking
#     High throughput, low latency, higher CPU usage.
#  - busy_spinning
#     Avoids syscalls which could introduce latency jitter. Best when threads can be bound to specific CPU
# During normal operation the journal will be smaller.
#message_journal_max_age = 12h
#message_journal_max_size = 5gb

#message_journal_flush_age = 1m
#message_journal_flush_interval = 1000000
#message_journal_segment_age = 1h
#message_journal_segment_size = 100mb

# Number of threads used exclusively for dispatching internal events. Default is 2.
#async_eventbus_processors = 2

# How many seconds to wait between marking node as DEAD for possible load balancers and starting the actual
# shutdown process. Set to 0 if you have no status checking load balancers in front.
lb_recognition_period_seconds = 3

# Journal usage percentage that triggers requesting throttling for this server node from load balancers. The feature is
# disabled if not set.
#lb_throttle_threshol

# Time in milliseconds after which a detected stale master node is being rechecked on startup.
#stale_master_timeout = 2000

# Time in milliseconds which Graylog is waiting for all threads to stop on shutdown.
#shutdown_timeout = 30000

# MongoDB connection string
# See https://docs.mongodb.com/manual/reference/connection-string/ for details
mongodb_uri = mongodb://localhost/graylog

# Authenticate against the MongoDB server
# '+'-signs in the username or password need to be replaced by '%2B'
mongodb_uri = mongodb://user:pwd@127.0.4.1:27017/graylog


# Use a replica set instead of a si
#    Using SMTP with STARTTLS *and* SMTPS at the same time is *not* possible.

# Use SMTP with STARTTLS, see https://en.wikipedia.org/wiki/Opportunistic_TLS
#transport_email_use_tls = true

# Use SMTP over SSL (SMTPS), see https://en.wikipedia.org/wiki/SMTPS
# This is deprecated on most SMTP services!
#transport_email_use_ssl = false


# Specify and uncomment this if you want to include links to the stream in your stream alert mails.
# This should define the fully qualified base url to your web interface exactly the same way as it is accessed by your users.
#transport_email_web_interface_url = https://graylog.example.com

# The default connect timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
# Default: 5s
#http_connect_timeout = 5s

# The
# Any host matching one of these patterns will be reached through a direct connection instead of through a proxy.
# Examples:
#   - localhost,127.0.0.1
#   - 10.0.*,*.example.com
#http_non_proxy_hosts =

# Disable the optimization of Elasticsearch indices after index cycling. This may take some load from Elasticsearch
# on heavily used systems with large indices, but it will decrease search performance. The default is to optimize
# cycled indices.
#
# ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
#            to your previous settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
#disable_index_optimization =
liseconds.
#ldap_connection_timeout = 2000

# Disable the use of SIGAR for collecting system stats
#disable_sigar = false

# The default cache time for dashboard widgets. (Default: 10 seconds, minimum: 1 second)
#dashboard_widget_default_cache_time = 10s

# For some cluster-related REST requests, the node must query all other nodes in the cluster. This is the maximum number
# of threads available for this. Increase it, if '/cluster/*' requests take long to complete.
# Should be http_thread_pool_size * average_cluster_size if you have a high number of concurrent users.
proxied_requests_thread_pool_size = 32

# The server is writing processing status information to the database on a regular basis. This setting controls how
# often the data is written to the database.
# Default: 1s (cannot be less than 1s)
#pro
# Default: 1m (one minute)
#processing_status_update_threshold = 1m

# Configures the journal write rate threshold for selecting processing status records. Any records that have a lower
# one minute rate than the configured value might be ignored. (dependent on number of messages in the journal)
# Default: 1
#processing_status_journal_write_rate_threshold = 1
17

Hello,

My apologies I forgot the quotes. Below is my lab configure.

grep -v "^#\|^$" /etc/graylog/server/server.conf | sed -e "s/#.*$//g"

[root@graylog server]# grep -v “^#|^$” /etc/graylog/server/server.conf | sed -e “s/#.*$//g”
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = string
root_password_sha2 =string
root_email = “greg.smith@domain.com”
root_timezone = Some/Place
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 10.10.10.10:9000
http_publish_uri = https://graylog.domain.com:9000/
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/ssl/certs/graylog/graylog-certificate.pem
http_tls_key_file = /etc/ssl/certs/graylog/graylog-key.pem
http_tls_key_password = secret
elasticsearch_hosts = http://10.10.10.10:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = true
allow_highlighting = false
elasticsearch_analyzer = standard
elasticsearch_index_optimization_timeout = 1h
output_batch_size = 5000
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 6
outputbuffer_processors = 2
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 3
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_size = 12gb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://mongo_admin:password123@localhost:27017/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
transport_email_enabled = true
transport_email_hostname = localhost
tansport_email_port = 25
transport_email_subject_prefix = [graylog]
transport_email_from_email = root@enseva-labs.net
transport_email_web_interface_url = https://10.10.10.10:9000
http_connect_timeout = 10s
proxied_requests_thread_pool_size = 32
[root@graylog server]#

Really not sure what you used to configured you Graylog server and applied settings. It seams like random configurations to me from what your showing us. Is there any more details you can provide?
Judging from my configurations, your Graylog configuration file looks like its missing a few configurations, are you sure this is the full file?

18

Yes, it is the whole config based on a sample conf file from ports/pkgs in FreeBSD.

I have now try to use your config with some changes in the next few days and come back with some feedback. Many thanks.

19

No luck Sirs. It continues to crash.

20

Hello,

To help you further we would need more information then

Like log files, perhaps tail’ing them when the service was restarted.
A full and completed configuration file would help not just part of it. Llike I showed you above.

grep -v "^#\|^$" /etc/graylog/server/server.conf | sed -e "s/#.*$//g"

Taking snap shot of any error/warnings on your WebUI helps a lot also.