Still working on graylog

[fffhurst (Franklin Hurst)]

This is a continuation of these posts:

Multiple problems with Graylog

1. Describe your incident: Graylog server was up and running and receiving logs, but not from certain hosts. All inputs show not running, but all inputs are receiving messages. All sidecar logs show x.509 certificate not valid error. Recently was able to restart sidecar on one of the vm’s sending logs, but a few days after this, the graylog web admin site began to intermittently become unavailable. If you reboot the server, it becomes available for a time, a few days, then it times out if y…

Certificate problem, and maybe other issues

Hello. This is a continuation of this thread since it got closed by the system: https://community.graylog.org/t/multiple-problems-with-graylog/22591 Based on the feedback from that thread I started looking into certificate issues and the most pressing issue was just getting the graylog web admin page to remain operational. In the test environment which an exact snapshot of production, I was able to update the PKI cert and have it signed by the CA. In the test, this had the following effects…

So recently the logs were giving message related to out of memory so I tried upping both the Elasticsearch and the Graylog heap memory settings to 4GB. I rebooted the server this morning, but the Grayog web admin page is still unavailable. Configuration is in the links above.

Graylog server logs from this morning:

2022-05-04T03:30:56.753-04:00 INFO  [AbstractRotationStrategy] Deflector index <Default index set> (index set <graylog_270>) should be rotated, Pointing deflector to new index now!
2022-05-04T03:30:56.755-04:00 INFO  [MongoIndexSet] Cycling from <graylog_270> to <graylog_271>.
2022-05-04T03:30:56.755-04:00 INFO  [MongoIndexSet] Creating target index <graylog_271>.
2022-05-04T03:30:56.903-04:00 INFO  [Indices] Successfully created index template graylog-internal
2022-05-04T03:30:57.156-04:00 INFO  [MongoIndexSet] Waiting for allocation of index <graylog_271>.
2022-05-04T03:30:57.248-04:00 INFO  [MongoIndexSet] Index <graylog_271> has been successfully allocated.
2022-05-04T03:30:57.248-04:00 INFO  [MongoIndexSet] Pointing index alias <graylog_deflector> to new index <graylog_271>.
2022-05-04T03:30:57.286-04:00 INFO  [SystemJobManager] Submitted SystemJob <23742440-cb7c-11ec-86a9-0050568bf8e7> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
2022-05-04T03:30:57.286-04:00 INFO  [MongoIndexSet] Successfully pointed index alias <graylog_deflector> to index <graylog_271>.
2022-05-04T03:31:27.309-04:00 INFO  [SetIndexReadOnlyJob] Flushing old index <graylog_270>.
2022-05-04T03:31:27.630-04:00 INFO  [SetIndexReadOnlyJob] Setting old index <graylog_270> to read-only.
2022-05-04T03:31:27.656-04:00 INFO  [SystemJobManager] Submitted SystemJob <358e8a80-cb7c-11ec-86a9-0050568bf8e7> [org.graylog2.indexer.indices.jobs.OptimizeIndexJob]
2022-05-04T03:31:27.671-04:00 INFO  [CreateNewSingleIndexRangeJob] Calculating ranges for index graylog_270.
2022-05-04T03:31:27.672-04:00 INFO  [OptimizeIndexJob] Optimizing index <graylog_270>.
2022-05-04T03:31:29.654-04:00 INFO  [MongoIndexRangeService] Calculated range of [graylog_270] in [1982ms].
2022-05-04T03:31:29.670-04:00 INFO  [CreateNewSingleIndexRangeJob] Created ranges for index graylog_270.
2022-05-04T03:31:29.681-04:00 INFO  [SystemJobManager] SystemJob <23742440-cb7c-11ec-86a9-0050568bf8e7> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob] finished in 2394ms.
2022-05-04T03:33:44.969-04:00 INFO  [SystemJobManager] SystemJob <358e8a80-cb7c-11ec-86a9-0050568bf8e7> [org.graylog2.indexer.indices.jobs.OptimizeIndexJob] finished in 137312ms.
2022-05-04T03:35:26.814-04:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Number of indices (21) higher than limit (20). Running retention for 1 indices.
2022-05-04T03:35:26.988-04:00 INFO  [AbstractIndexCountBasedRetentionStrategy] Running retention strategy [org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy] for index <graylog_251>
2022-05-04T03:35:27.246-04:00 INFO  [DeletionRetentionStrategy] Finished index retention strategy [delete] for index <graylog_251> in 247ms.
2022-05-04T05:32:08.405-04:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0x961dbf2e, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:54339]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)
2022-05-04T05:32:38.492-04:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0xccfc57da, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:55002]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)
2022-05-04T05:34:50.296-04:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0xa6b916d7, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:57524]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)
2022-05-04T05:36:02.334-04:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0x0a4f1e4f, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:60046]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)
2022-05-04T05:36:20.296-04:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0x3799f3a2, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:63765]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)
2022-05-04T05:36:46.379-04:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0xcd043d3e, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:49783]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)
2022-05-04T05:38:23.485-04:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0xaa71240f, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:52629]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)
2022-05-04T05:39:37.171-04:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0xa2726f67, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:56902]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)
2022-05-04T05:40:25.328-04:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0x9e7387bd, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:57839]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)
2022-05-04T05:40:31.344-04:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0xe3cc594e, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:57875]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)
2022-05-04T05:40:42.563-04:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0x5eda44fd, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:57949]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)
2022-05-04T05:40:48.578-04:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0xe5bcc0e8, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:57991]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)
2022-05-04T05:41:41.109-04:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0xf9d10a6e, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:58702]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)
2022-05-04T05:42:12.642-04:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0xdaf77839, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:59094]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)
2022-05-04T05:42:20.791-04:00 ERROR [AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0xda39ad9e, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:59145]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)
2022-05-04T08:49:24.268-04:00 INFO  [Server] SIGNAL received. Shutting down.
2022-05-04T08:50:21.822-04:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 3.0.2 [org.graylog.aws.AWSPlugin]
2022-05-04T08:50:21.844-04:00 INFO  [CmdLineTool] Loaded plugin: Collector 3.0.2 [org.graylog.plugins.collector.CollectorPlugin]
2022-05-04T08:50:21.845-04:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 3.0.2 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2022-05-04T08:50:22.639-04:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms4g -Xmx4g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.truststore=/SSCLant/certs/cacerts.jks -Dlog4j2.formatMsgNoLookups=true -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=rpm
2022-05-04T08:50:23.187-04:00 INFO  [Version] HV000001: Hibernate Validator 5.1.3.Final
2022-05-04T08:50:26.433-04:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-05-04T08:50:26.481-04:00 INFO  [NodeId] Node ID: 9e804424-96fe-42c8-a79a-051fc7fb0963
2022-05-04T08:50:26.828-04:00 INFO  [LogManager] Loading logs.
2022-05-04T08:50:26.925-04:00 WARN  [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000005922176614.index, deleting and rebuilding index...
2022-05-04T08:50:27.426-04:00 INFO  [LogManager] Logs loading complete.
2022-05-04T08:50:27.432-04:00 INFO  [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2022-05-04T08:50:27.481-04:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2022-05-04T08:50:27.532-04:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2022-05-04T08:50:27.621-04:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-05-04T08:50:27.954-04:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:1}] to localhost:27017
2022-05-04T08:50:27.969-04:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 9]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=12950285}
2022-05-04T08:50:28.098-04:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:2}] to localhost:27017
2022-05-04T08:50:29.056-04:00 INFO  [AbstractJestClient] Setting server pool to a list of 1 servers: [http://127.0.0.1:9200]
2022-05-04T08:50:29.057-04:00 INFO  [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2022-05-04T08:50:29.314-04:00 INFO  [JestClientFactory] Using custom ObjectMapper instance
2022-05-04T08:50:29.315-04:00 INFO  [JestClientFactory] Node Discovery disabled...
2022-05-04T08:50:29.315-04:00 INFO  [JestClientFactory] Idle connection reaping disabled...
2022-05-04T08:50:29.558-04:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-05-04T08:50:32.763-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-05-04T08:50:32.807-04:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-05-04T08:50:32.906-04:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:3}] to localhost:27017
2022-05-04T08:50:34.497-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-05-04T08:50:36.228-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-05-04T08:50:38.074-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-05-04T08:50:39.597-04:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-05-04T08:50:40.324-04:00 INFO  [ServerBootstrap] Graylog server 3.0.2+1686930 starting up
2022-05-04T08:50:40.324-04:00 INFO  [ServerBootstrap] JRE: Red Hat, Inc. 1.8.0_282 on Linux 3.10.0-1160.15.2.el7.x86_64
2022-05-04T08:50:40.324-04:00 INFO  [ServerBootstrap] Deployment: rpm
2022-05-04T08:50:40.324-04:00 INFO  [ServerBootstrap] OS: CentOS Linux 7 (Core) (centos)
2022-05-04T08:50:40.325-04:00 INFO  [ServerBootstrap] Arch: amd64
2022-05-04T08:50:40.385-04:00 INFO  [PeriodicalsService] Starting 27 periodicals ...
2022-05-04T08:50:40.386-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2022-05-04T08:50:40.402-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2022-05-04T08:50:40.403-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2022-05-04T08:50:40.416-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2022-05-04T08:50:40.418-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2022-05-04T08:50:40.426-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2022-05-04T08:50:40.427-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2022-05-04T08:50:40.428-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2022-05-04T08:50:40.428-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2022-05-04T08:50:40.446-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2022-05-04T08:50:40.446-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2022-05-04T08:50:40.447-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2022-05-04T08:50:40.448-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2022-05-04T08:50:40.449-04:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2022-05-04T08:50:40.449-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2022-05-04T08:50:40.449-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2022-05-04T08:50:40.454-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2022-05-04T08:50:40.471-04:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2022-05-04T08:50:40.532-04:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:4}] to localhost:27017
2022-05-04T08:50:40.541-04:00 INFO  [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2022-05-04T08:50:40.541-04:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:5}] to localhost:27017
2022-05-04T08:50:40.541-04:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:7}] to localhost:27017
2022-05-04T08:50:40.541-04:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:6}] to localhost:27017
2022-05-04T08:50:40.579-04:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2022-05-04T08:50:40.579-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2022-05-04T08:50:40.591-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2022-05-04T08:50:40.643-04:00 ERROR [Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)
2022-05-04T08:50:40.643-04:00 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2022-05-04T08:50:40.649-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2022-05-04T08:50:40.661-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2022-05-04T08:50:40.661-04:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2022-05-04T08:50:40.662-04:00 INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2022-05-04T08:50:40.662-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2022-05-04T08:50:40.665-04:00 INFO  [IndexFieldTypePollerPeriodical] Cluster not connected yet, delaying index field type initialization until it is reachable.
2022-05-04T08:50:40.666-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2022-05-04T08:50:40.667-04:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2022-05-04T08:50:40.857-04:00 INFO  [V20161130141500_DefaultStreamRecalcIndexRanges] Cluster not connected yet, delaying migration until it is reachable.
2022-05-04T08:50:41.005-04:00 INFO  [JerseyService] Enabling CORS for HTTP endpoint
2022-05-04T08:50:55.468-04:00 INFO  [IndexRangesCleanupPeriodical] Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy
2022-05-04T08:51:05.742-04:00 INFO  [NetworkListener] Started listener bound to [<graylogserverFQDN>:9000]
2022-05-04T08:51:05.744-04:00 INFO  [HttpServer] [HttpServer] Started.
2022-05-04T08:51:05.744-04:00 INFO  [JerseyService] Started REST API at <<graylogserverFQDN>:9000>
2022-05-04T08:51:05.756-04:00 INFO  [ServiceManagerListener] Services are healthy
2022-05-04T08:51:05.757-04:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2022-05-04T08:51:05.759-04:00 INFO  [ServerBootstrap] Services started, startup times in ms: {ConfigurationEtagService [RUNNING]=78, OutputSetupService [RUNNING]=80, EtagService [RUNNING]=80, JournalReader [RUNNING]=83, GracefulShutdownService [RUNNING]=84, InputSetupService [RUNNING]=87, BufferSynchronizerService [RUNNING]=167, KafkaJournal [RUNNING]=202, PeriodicalsService [RUNNING]=333, LookupTableService [RUNNING]=334, StreamCacheService [RUNNING]=376, JerseyService [RUNNING]=25379}
2022-05-04T08:51:05.900-04:00 INFO  [ServerBootstrap] Graylog server up and running.
2022-05-04T08:51:06.060-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5cf1398829fbc65472f9e760] is now STARTING
2022-05-04T08:51:06.082-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5cf13c1329fbc65472f9ea2a] is now STARTING
2022-05-04T08:51:06.139-04:00 INFO  [InputStateListener] Input [GELF TCP/5cf13a5629fbc65472f9e843] is now STARTING
2022-05-04T08:51:06.141-04:00 INFO  [InputStateListener] Input [Syslog UDP/5cf13c4629fbc65472f9ea68] is now STARTING
2022-05-04T08:51:06.173-04:00 INFO  [InputStateListener] Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] is now STARTING
2022-05-04T08:51:06.308-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x9dbb0a63, L:/0.0.0.0:12201]) should be 262144 but is 425984.
2022-05-04T08:51:06.310-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xca27645b, L:/0.0.0.0:5140]) should be 262144 but is 425984.
2022-05-04T08:51:06.444-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xb9fe2c0d, L:/0.0.0.0:1514]) should be 262144 but is 425984.
2022-05-04T08:51:06.617-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x885ce160, L:/0.0.0.0:5140]) should be 262144 but is 425984.
2022-05-04T08:51:06.617-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x0a6ab708, L:/0.0.0.0:12201]) should be 262144 but is 425984.
2022-05-04T08:51:06.809-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x81e834c8, L:/0.0.0.0:1514]) should be 262144 but is 425984.
2022-05-04T08:51:06.811-04:00 WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input GELFTCPInput{title=Windows Event Logs, type=org.graylog2.inputs.gelf.tcp.GELFTCPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x9dcccd31, L:/0.0.0.0:12201]) should be 1048576 but is 425984.
2022-05-04T08:51:06.814-04:00 INFO  [InputStateListener] Input [GELF TCP/5cf13a5629fbc65472f9e843] is now RUNNING
2022-05-04T08:51:06.896-04:00 WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=COC Infrastructure, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x62c7633f, L:/0.0.0.0:1514]) should be 1048576 but is 425984.
2022-05-04T08:51:06.897-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x54e777c3, L:/0.0.0.0:5140]) should be 262144 but is 425984.
2022-05-04T08:51:06.899-04:00 INFO  [InputStateListener] Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] is now RUNNING
2022-05-04T08:51:06.922-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x96babe6d, L:/0.0.0.0:12201]) should be 262144 but is 425984.
2022-05-04T08:51:06.925-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xf426d4d2, L:/0.0.0.0:1514]) should be 262144 but is 425984.
2022-05-04T08:51:06.973-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=FirePOWER, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x5c907757, L:/0.0.0.0:5140]) should be 262144 but is 425984.
2022-05-04T08:51:06.978-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Firewall, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0x5ccdc445, L:/0.0.0.0:1514]) should be 262144 but is 425984.
2022-05-04T08:51:06.979-04:00 WARN  [UdpTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=APC Logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=9e804424-96fe-42c8-a79a-051fc7fb0963} (channel [id: 0xbedaf1ce, L:/0.0.0.0:12201]) should be 262144 but is 425984.
2022-05-04T08:51:06.999-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5cf13c1329fbc65472f9ea2a] is now RUNNING
2022-05-04T08:51:07.003-04:00 INFO  [InputStateListener] Input [Syslog UDP/5cf13c4629fbc65472f9ea68] is now RUNNING
2022-05-04T08:51:07.004-04:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5cf1398829fbc65472f9e760] is now RUNNING

So at this point, just trying to get the graylog web admin page up and running.

[gsmith]

Hello,

You have a lot going on, I realize your fairly new to this software. What I can do is break it down from the logs that are shown above.

This error below shows that something is wrong with your INPUT “Syslog.TCP/5cf13c2e29fbc65472f9ea4a”.

• So how did you configure this input and what have you done to resolve this issue?

[AbstractTcpTransport] Error in Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] (channel [id: 0x961dbf2e, L:/<graylogserverIP>:1514 ! R:/<ScannerIP2>:54339]) (cause io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer)

This Warning below shows there was something wrong with your index and graylog tried to correct it. I personally would use Curl commands from the other post/s to see if you shards are good.

Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000005922176614.index, deleting and rebuilding index

• I realize it is a tedious task, But could you show your current configurations and versions here?

This error below shows that Graylog could not connect to elasticsearch.

• Could you show how you configured your indices and configuration to Elasticsearch?

Cluster] Couldn't read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)

What I noticed was this in the logs, It seams that all the input are running. I’m assuming this was a restart of the node or services on Graylog. Sometimes these errors occur because the incorrect service was start first. Example: Elasticsearch and MongoDb should be running before Graylog.

INFO  [InputStateListener] Input [Syslog TCP/5cf13c2e29fbc65472f9ea4a] is now RUNNING
INFO  [InputStateListener] Input [Raw/Plaintext UDP/5cf13c1329fbc65472f9ea2a] is now RUNNING
INFO  [InputStateListener] Input [Syslog UDP/5cf13c4629fbc65472f9ea68] is now RUNNING
INFO  [InputStateListener] Input [Raw/Plaintext UDP/5cf1398829fbc65472f9e760] is now RUNNING

If your using certificates I would fall back to just using http to see if that is the problem.
Start small and work you way up is the best way to troubleshoot major issues with multiple problems.

These issue/s could be multiple things gone wrong. For example configuration between MongoDb , Elasticsearch and Graylog. Something is preventing Graylog from connecting to Elasticsearch. Using certificates that are incorrect or not configured properly with Graylog. The certification issue is a major problem I have seen here. 9 /10 times when members use certificates and have issues it because of that.

[fffhurst (Franklin Hurst)]

@gsmith

This error below shows that something is wrong with your INPUT “Syslog.TCP/5cf13c2e29fbc65472f9ea4a”.

• So how did you configure this input and what have you done to resolve this issue?

I did not configure the input. I am picking up someone else’s work. The graylog web admin page is down, so I do not know how to reconfigure the input without getting into the web admin page.

[fffhurst (Franklin Hurst)]

@gsmith

I realize it is a tedious task, But could you show your current configurations and versions here?

Feb 8

2. Describe your environment:

• OS Information:
  Sidecars are on Windows. Graylog server is on CentOS7.
• Package Version:
  graylog-server-3.0.2-1.noarch
  elasticsearch-6.8.0-1.noarch
  mongodb-org-tools-4.0.9-1.el7.x86_64
  monbodb-org-server-4.0.9-1.el7.x86_64
  mongodb-org-mongos-4.0.9-1.el7.x86_64
  mongodb-org-shell-4.0.9-1.el7.x86_64

[fffhurst (Franklin Hurst)]

@gsmith

I realize it is a tedious task, But could you show your current configurations and versions here?

############################
# GRAYLOG CONFIGURATION FILE
############################


# (deleted initial explanatory part of file)

# If you are running more than one instances of Graylog server you have to select one of these
# instances as master. The master will perform some periodical tasks that non-masters won't perform.
is_master = true

# The auto-generated node ID will be stored in this file and read after restarts. It is a good idea
# to use an absolute file path here if you are starting Graylog server from init scripts or similar.
node_id_file = /etc/graylog/server/node-id

# You MUST set a secret to secure/pepper the stored user passwords here. Use at least 64 characters.
# Generate one by using for example: pwgen -N 1 -s 96
password_secret = <removed secret>

# The default root user is named 'admin'
root_username = root

# You MUST specify a hash password for the root user (which you only need to initially set up the
# system and in case you lose connectivity to your authentication backend)
# This password cannot be changed using the API or via the web interface. If you need to change it,
# modify it in this file.
# Create one by using for example: echo -n yourpassword | shasum -a 256
# and put the resulting hash value into the following line
root_password_sha2 = <removed password>

# The email address of the root user.
# Default is empty
#root_email = ""

# The time zone setting of the root user. See http://www.joda.org/joda-time/timezones.html for a list of valid time zones.
# Default is UTC
#root_timezone = UTC

# Set the bin directory here (relative or absolute)
# This directory contains binaries that are used by the Graylog server.
# Default: bin
bin_dir = /usr/share/graylog-server/bin

# Set the data directory here (relative or absolute)
# This directory is used to store Graylog server state.
# Default: data
data_dir = /var/lib/graylog-server

# Set plugin directory here (relative or absolute)
plugin_dir = /usr/share/graylog-server/plugin

###############
# HTTP settings
###############

#### HTTP bind address
#
# The network interface used by the Graylog HTTP interface.
#
# This network interface must be accessible by all Graylog nodes in the cluster and by all clients
# using the Graylog web interface.
#
# If the port is omitted, Graylog will use port 9000 by default.
#
# Default: 127.0.0.1:9000
http_bind_address = <removed http_bind_address>

#### HTTP publish URI
#
# The HTTP URI of this Graylog node which is used to communicate with the other Graylog nodes in the cluster and by all
# clients using the Graylog web interface.
#
# The URI will be published in the cluster discovery APIs, so that other Graylog nodes will be able to find and connect to this Graylog node.
#
# This configuration setting has to be used if this Graylog node is available on another network interface than $http_bind_address,
# for example if the machine has multiple network interfaces or is behind a NAT gateway.
#
# If $http_bind_address contains a wildcard IPv4 address (0.0.0.0), the first non-loopback IPv4 address of this machine will be used.
# This configuration setting *must not* contain a wildcard address!
#
# Default: http://$http_bind_address/
http_publish_uri = <removed URI>

#### External Graylog URI
#
# The public URI of Graylog which will be used by the Graylog web interface to communicate with the Graylog REST API.
#
# The external Graylog URI usually has to be specified, if Graylog is running behind a reverse proxy or load-balancer
# and it will be used to generate URLs addressing entities in the Graylog REST API (see $http_bind_address).
#
# When using Graylog Collector, this URI will be used to receive heartbeat messages and must be accessible for all collectors.
#
# This setting can be overriden on a per-request basis with the "X-Graylog-Server-URL" HTTP request header.
#
# Default: $http_publish_uri
#http_external_uri =

#### Enable CORS headers for HTTP interface
#
# This is necessary for JS-clients accessing the server directly.
# If these are disabled, modern browsers will not be able to retrieve resources from the server.
# This is enabled by default. Uncomment the next line to disable it.
#http_enable_cors = false

#### Enable GZIP support for HTTP interface
#
# This compresses API responses and therefore helps to reduce
# overall round trip times. This is enabled by default. Uncomment the next line to disable it.
#http_enable_gzip = false

# The maximum size of the HTTP request headers in bytes.
#http_max_header_size = 8192

# The size of the thread pool used exclusively for serving the HTTP interface.
#http_thread_pool_size = 16

################
# HTTPS settings
################

#### Enable HTTPS support for the HTTP interface
#
# This secures the communication with the HTTP interface with TLS to prevent request forgery and eavesdropping.
#
# Default: false
http_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the HTTP interface.
http_tls_cert_file = <removed cert file location>

# The PKCS#8 private key file in PEM format to use for securing the HTTP interface.
http_tls_key_file = <removed key location>

# The password to unlock the private key used for securing the HTTP interface.
http_tls_key_password = <removed password>

# Comma separated list of trusted proxies that are allowed to set the client address with X-Forwarded-For
# header. May be subnets, or hosts.
#trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128

# List of Elasticsearch hosts Graylog should connect to.
# Need to be specified as a comma-separated list of valid URIs for the http ports of your elasticsearch nodes.
# If one or more of your elasticsearch hosts require authentication, include the credentials in each node URI that
# requires authentication.
#
# Default: http://127.0.0.1:9200
#elasticsearch_hosts = http://node1:9200,http://user:password@node2:19200

# Maximum amount of time to wait for successfull connection to Elasticsearch HTTP port.
#
# Default: 10 Seconds
#elasticsearch_connect_timeout = 10s

# Maximum amount of time to wait for reading back a response from an Elasticsearch server.
#
# Default: 60 seconds
#elasticsearch_socket_timeout = 60s

# Maximum idle time for an Elasticsearch connection. If this is exceeded, this connection will
# be tore down.
#
# Default: inf
#elasticsearch_idle_timeout = -1s

# Maximum number of total connections to Elasticsearch.
#
# Default: 20
#elasticsearch_max_total_connections = 20

# Maximum number of total connections per Elasticsearch route (normally this means per
# elasticsearch server).
#
# Default: 2
#elasticsearch_max_total_connections_per_route = 2

# Maximum number of times Graylog will retry failed requests to Elasticsearch.
#
# Default: 2
#elasticsearch_max_retries = 2

# Enable automatic Elasticsearch node discovery through Nodes Info,
# see https://www.elastic.co/guide/en/elasticsearch/reference/5.4/cluster-nodes-info.html
#
# WARNING: Automatic node discovery does not work if Elasticsearch requires authentication, e. g. with Shield.
#
# Default: false
#elasticsearch_discovery_enabled = true

# Filter for including/excluding Elasticsearch nodes in discovery according to their custom attributes,
# see https://www.elastic.co/guide/en/elasticsearch/reference/5.4/cluster.html#cluster-nodes
#
# Default: empty
#elasticsearch_discovery_filter = rack:42

# Frequency of the Elasticsearch node discovery.
#
# Default: 30s
# elasticsearch_discovery_frequency = 30s

# Enable payload compression for Elasticsearch requests.
#
# Default: false
#elasticsearch_compression_enabled = true

# Graylog will use multiple indices to store documents in. You can configured the strategy it uses to determine
# when to rotate the currently active write index.
# It supports multiple rotation strategies:
#   - "count" of messages per index, use elasticsearch_max_docs_per_index below to configure
#   - "size" per index, use elasticsearch_max_size_per_index below to configure
# valid values are "count", "size" and "time", default is "count"
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
rotation_strategy = count

# (Approximate) maximum number of documents in an Elasticsearch index before a new index
# is being created, also see no_retention and elasticsearch_max_number_of_indices.
# Configure this if you used 'rotation_strategy = count' above.
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
elasticsearch_max_docs_per_index = 20000000

# (Approximate) maximum size in bytes per Elasticsearch index on disk before a new index is being created, also see
# no_retention and elasticsearch_max_number_of_indices. Default is 1GB.
# Configure this if you used 'rotation_strategy = size' above.
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
#elasticsearch_max_size_per_index = 1073741824

# (Approximate) maximum time before a new Elasticsearch index is being created, also see
# no_retention and elasticsearch_max_number_of_indices. Default is 1 day.
# Configure this if you used 'rotation_strategy = time' above.
# Please note that this rotation period does not look at the time specified in the received messages, but is
# using the real clock value to decide when to rotate the index!
# Specify the time using a duration and a suffix indicating which unit you want:
#  1w  = 1 week
#  1d  = 1 day
#  12h = 12 hours
# Permitted suffixes are: d for day, h for hour, m for minute, s for second.
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
#elasticsearch_max_time_per_index = 1d

# Disable checking the version of Elasticsearch for being compatible with this Graylog release.
# WARNING: Using Graylog with unsupported and untested versions of Elasticsearch may lead to data loss!
#elasticsearch_disable_version_check = true

# Disable message retention on this node, i. e. disable Elasticsearch index rotation.
#no_retention = false

# How many indices do you want to keep?
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
elasticsearch_max_number_of_indices = 20

# Decide what happens with the oldest indices when the maximum number of indices is reached.
# The following strategies are availble:
#   - delete # Deletes the index completely (Default)
#   - close # Closes the index and hides it from the system. Can be re-opened later.
#
# ATTENTION: These settings have been moved to the database in 2.0. When you upgrade, make sure to set these
#            to your previous 1.x settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
retention_strategy = delete

# How many Elasticsearch shards and replicas should be used per index? Note that this only applies to newly created indices.
# ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
#            to your previous settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
elasticsearch_shards = 4
elasticsearch_replicas = 0

# Prefix for all Elasticsearch indices and index aliases managed by Graylog.
#
# ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
#            to your previous settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
elasticsearch_index_prefix = graylog

# Name of the Elasticsearch index template used by Graylog to apply the mandatory index mapping.
# Default: graylog-internal
#
# ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
#            to your previous settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
#elasticsearch_template_name = graylog-internal

# Do you want to allow searches with leading wildcards? This can be extremely resource hungry and should only
# be enabled with care. See also: http://docs.graylog.org/en/2.1/pages/queries.html
allow_leading_wildcard_searches = false

# Do you want to allow searches to be highlighted? Depending on the size of your messages this can be memory hungry and
# should only be enabled after making sure your Elasticsearch cluster has enough memory.
allow_highlighting = false

# Analyzer (tokenizer) to use for message and full_message field. The "standard" filter usually is a good idea.
# All supported analyzers are: standard, simple, whitespace, stop, keyword, pattern, language, snowball, custom
# Elasticsearch documentation: https://www.elastic.co/guide/en/elasticsearch/reference/2.3/analysis.html
# Note that this setting only takes effect on newly created indices.
#
# ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
#            to your previous settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
elasticsearch_analyzer = standard

# Global request timeout for Elasticsearch requests (e. g. during search, index creation, or index time-range
# calculations) based on a best-effort to restrict the runtime of Elasticsearch operations.
# Default: 1m
#elasticsearch_request_timeout = 1m

# Global timeout for index optimization (force merge) requests.
# Default: 1h
#elasticsearch_index_optimization_timeout = 1h

# Maximum number of concurrently running index optimization (force merge) jobs.
# If you are using lots of different index sets, you might want to increase that number.
# Default: 20
#elasticsearch_index_optimization_jobs = 20

# Time interval for index range information cleanups. This setting defines how often stale index range information
# is being purged from the database.
# Default: 1h
#index_ranges_cleanup_interval = 1h

# Time interval for the job that runs index field type maintenance tasks like cleaning up stale entries. This doesn't
# need to run very often.
# Default: 1h
#index_field_type_periodical_interval = 1h

# Batch size for the Elasticsearch output. This is the maximum (!) number of messages the Elasticsearch output
# module will get at once and write to Elasticsearch in a batch call. If the configured batch size has not been
# reached within output_flush_interval seconds, everything that is available will be flushed at once. Remember
# that every outputbuffer processor manages its own batch and performs its own batch write calls.
# ("outputbuffer_processors" variable)
output_batch_size = 500

# Flush interval (in seconds) for the Elasticsearch output. This is the maximum amount of time between two
# batches of messages written to Elasticsearch. It is only effective at all if your minimum number of messages
# for this time period is less than output_batch_size * outputbuffer_processors.
output_flush_interval = 1

# As stream outputs are loaded only on demand, an output which is failing to initialize will be tried over and
# over again. To prevent this, the following configuration options define after how many faults an output will
# not be tried again for an also configurable amount of seconds.
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30

# The number of parallel running processors.
# Raise this number if your buffers are filling up.
processbuffer_processors = 5
outputbuffer_processors = 3

# The following settings (outputbuffer_processor_*) configure the thread pools backing each output buffer processor.
# See https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadPoolExecutor.html for technical details

# When the number of threads is greater than the core (see outputbuffer_processor_threads_core_pool_size),
# this is the maximum time in milliseconds that excess idle threads will wait for new tasks before terminating.
# Default: 5000
#outputbuffer_processor_keep_alive_time = 5000

# The number of threads to keep in the pool, even if they are idle, unless allowCoreThreadTimeOut is set
# Default: 3
#outputbuffer_processor_threads_core_pool_size = 3

# The maximum number of threads to allow in the pool
# Default: 30
#outputbuffer_processor_threads_max_pool_size = 30

# UDP receive buffer size for all message inputs (e. g. SyslogUDPInput).
#udp_recvbuffer_sizes = 1048576

# Wait strategy describing how buffer processors wait on a cursor sequence. (default: sleeping)
# Possible types:
#  - yielding
#     Compromise between performance and CPU usage.
#  - sleeping
#     Compromise between performance and CPU usage. Latency spikes can occur after quiet periods.
#  - blocking
#     High throughput, low latency, higher CPU usage.
#  - busy_spinning
#     Avoids syscalls which could introduce latency jitter. Best when threads can be bound to specific CPU cores.
processor_wait_strategy = blocking

# Size of internal ring buffers. Raise this if raising outputbuffer_processors does not help anymore.
# For optimum performance your LogMessage objects in the ring buffer should fit in your CPU L3 cache.
# Must be a power of 2. (512, 1024, 2048, ...)
ring_size = 65536

inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking

# Enable the disk based message journal.
message_journal_enabled = true

# The directory which will be used to store the message journal. The directory must me exclusively used by Graylog and
# must not contain any other files than the ones created by Graylog itself.
#
# ATTENTION:
#   If you create a seperate partition for the journal files and use a file system creating directories like 'lost+found'
#   in the root directory, you need to create a sub directory for your journal.
#   Otherwise Graylog will log an error message that the journal is corrupt and Graylog will not start.
message_journal_dir = /var/lib/graylog-server/journal

# Journal hold messages before they could be written to Elasticsearch.
# For a maximum of 12 hours or 5 GB whichever happens first.
# During normal operation the journal will be smaller.
#message_journal_max_age = 12h
#message_journal_max_size = 5gb

#message_journal_flush_age = 1m
#message_journal_flush_interval = 1000000
#message_journal_segment_age = 1h
#message_journal_segment_size = 100mb

# Number of threads used exclusively for dispatching internal events. Default is 2.
#async_eventbus_processors = 2

# How many seconds to wait between marking node as DEAD for possible load balancers and starting the actual
# shutdown process. Set to 0 if you have no status checking load balancers in front.
lb_recognition_period_seconds = 3

# Journal usage percentage that triggers requesting throttling for this server node from load balancers. The feature is
# disabled if not set.
#lb_throttle_threshold_percentage = 95

# Every message is matched against the configured streams and it can happen that a stream contains rules which
# take an unusual amount of time to run, for example if its using regular expressions that perform excessive backtracking.
# This will impact the processing of the entire server. To keep such misbehaving stream rules from impacting other
# streams, Graylog limits the execution time for each stream.
# The default values are noted below, the timeout is in milliseconds.
# If the stream matching for one stream took longer than the timeout value, and this happened more than "max_faults" times
# that stream is disabled and a notification is shown in the web interface.
#stream_processing_timeout = 2000
#stream_processing_max_faults = 3

# Length of the interval in seconds in which the alert conditions for all streams should be checked
# and alarms are being sent.
#alert_check_interval = 60

# Since 0.21 the Graylog server supports pluggable output modules. This means a single message can be written to multiple
# outputs. The next setting defines the timeout for a single output module, including the default output module where all
# messages end up.
#
# Time in milliseconds to wait for all message outputs to finish writing a single message.
#output_module_timeout = 10000

# Time in milliseconds after which a detected stale master node is being rechecked on startup.
#stale_master_timeout = 2000

# Time in milliseconds which Graylog is waiting for all threads to stop on shutdown.
#shutdown_timeout = 30000

# MongoDB connection string
# See https://docs.mongodb.com/manual/reference/connection-string/ for details
mongodb_uri = mongodb://mongo_admin:<removed connection string>

# Authenticate against the MongoDB server
# '+'-signs in the username or password need to be replaced by '%2B'
mongodb_uri = mongodb://mongo_admin:<removed>

# Use a replica set instead of a single host
mongodb_uri = mongodb://mongo_admin:<removed>

# Increase this value according to the maximum connections your MongoDB server can handle from a single client
# if you encounter MongoDB connection problems.
mongodb_max_connections = 1000

# Number of threads allowed to be blocked by MongoDB connections multiplier. Default: 5
# If mongodb_max_connections is 100, and mongodb_threads_allowed_to_block_multiplier is 5,
# then 500 threads can block. More than that and an exception will be thrown.
# http://api.mongodb.com/java/current/com/mongodb/MongoOptions.html#threadsAllowedToBlockForConnectionMultiplier
mongodb_threads_allowed_to_block_multiplier = 5


# Email transport
transport_email_enabled = true
transport_email_hostname = <removed>
transport_email_port = 25
transport_email_use_auth = false
#transport_email_auth_username = 
#transport_email_auth_password =
transport_email_subject_prefix = [graylog]
transport_email_from_email = <removed>

# Encryption settings
#
# ATTENTION:
#    Using SMTP with STARTTLS *and* SMTPS at the same time is *not* possible.

# Use SMTP with STARTTLS, see https://en.wikipedia.org/wiki/Opportunistic_TLS
transport_email_use_tls = false

# Use SMTP over SSL (SMTPS), see https://en.wikipedia.org/wiki/SMTPS
# This is deprecated on most SMTP services!
#transport_email_use_ssl = false


# Specify and uncomment this if you want to include links to the stream in your stream alert mails.
# This should define the fully qualified base url to your web interface exactly the same way as it is accessed by your users.
transport_email_web_interface_url = <removed>

# The default connect timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
# Default: 5s
#http_connect_timeout = 5s

# The default read timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
# Default: 10s
#http_read_timeout = 10s

# The default write timeout for outgoing HTTP connections.
# Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
# Default: 10s
#http_write_timeout = 10s

# HTTP proxy for outgoing HTTP connections
# ATTENTION: If you configure a proxy, make sure to also configure the "http_non_proxy_hosts" option so internal
#            HTTP connections with other nodes does not go through the proxy.
# Examples:
#   - http://proxy.example.com:8123
#   - http://username:password@proxy.example.com:8123
#http_proxy_uri =

# A list of hosts that should be reached directly, bypassing the configured proxy server.
# This is a list of patterns separated by ",". The patterns may start or end with a "*" for wildcards.
# Any host matching one of these patterns will be reached through a direct connection instead of through a proxy.
# Examples:
#   - localhost,127.0.0.1
#   - 10.0.*,*.example.com
#http_non_proxy_hosts =

# Disable the optimization of Elasticsearch indices after index cycling. This may take some load from Elasticsearch
# on heavily used systems with large indices, but it will decrease search performance. The default is to optimize
# cycled indices.
#
# ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
#            to your previous settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
#disable_index_optimization = true

# Optimize the index down to <= index_optimization_max_num_segments. A higher number may take some load from Elasticsearch
# on heavily used systems with large indices, but it will decrease search performance. The default is 1.
#
# ATTENTION: These settings have been moved to the database in Graylog 2.2.0. When you upgrade, make sure to set these
#            to your previous settings so they will be migrated to the database!
#            This configuration setting is only used on the first start of Graylog. After that,
#            index related settings can be changed in the Graylog web interface on the 'System / Indices' page.
#            Also see http://docs.graylog.org/en/2.3/pages/configuration/index_model.html#index-set-configuration.
#index_optimization_max_num_segments = 1

# The threshold of the garbage collection runs. If GC runs take longer than this threshold, a system notification
# will be generated to warn the administrator about possible problems with the system. Default is 1 second.
#gc_warning_threshold = 1s

# Connection timeout for a configured LDAP server (e. g. ActiveDirectory) in milliseconds.
#ldap_connection_timeout = 2000

# Disable the use of SIGAR for collecting system stats
#disable_sigar = false

# The default cache time for dashboard widgets. (Default: 10 seconds, minimum: 1 second)
#dashboard_widget_default_cache_time = 10s

# For some cluster-related REST requests, the node must query all other nodes in the cluster. This is the maximum number
# of threads available for this. Increase it, if '/cluster/*' requests take long to complete.
# Should be http_thread_pool_size * average_cluster_size if you have a high number of concurrent users.
proxied_requests_thread_pool_size = 32

[gsmith]

Hello,

@fffhurst

The graylog web admin page is down, so I do not know how to reconfigure the input without getting into the web admin page.

When you try to login the Web UI. what error/s are you seeing that’s preventing you from logging into the Web interface? I don’t see any information shown here that’s preventing a user from logging in to this Graylog instance.

What Have you tried to do so far to correct this issue?

For example what I stated earlier.

@gsmith

If your using certificates I would fall back to just using http to see if that is the problem.
Start small and work you way up is the best way to troubleshoot major issues with multiple problems.

What is the status of Elasticsearch, Graylog & MongoDb?
Are you tail’ing your Graylog file when trying to logging the Web interface? if so, what do you see?
Normally if there is a issue with logging into the Web interface this will be shown in Graylog log file.

[gsmith]

I found something here in you configuration file that does not seem correct. Only one should be configured, not three. If you using a user name and password on MongoDb make sure the credentials match for the user you add to MongoDb. If you did not create a user/password for MongoDb then I would just leave it as default

Example

mongodb_uri = mongodb://mongo_admin:some_password@localhost:27017/graylog

OR

mongodb_uri = mongodb://localhost:27017/graylog

@fffhurst

# MongoDB connection string
# See https://docs.mongodb.com/manual/reference/connection-string/ for details
mongodb_uri = mongodb://mongo_admin:<removed connection string>

# Authenticate against the MongoDB server
# '+'-signs in the username or password need to be replaced by '%2B'
mongodb_uri = mongodb://mongo_admin:<removed>

# Use a replica set instead of a single host
mongodb_uri = mongodb://mongo_admin:<removed>

EDIT:
I’m not sure if your HTTPS configuration is correct but here is mine, Please noticed the little details.
My certificates match my FQDN, so I matched my setting to correspond to those certs.

http_bind_address = graylog.domain.com:9000  <--- You can use an IP Address, I just prefer FQDN
http_publish_uri = https://graylog.domain.com:9000/  <--  HTTPS & FQDN or IPADDRESS
http_enable_cors = true
http_enable_tls = true
http_tls_cert_file = /etc/graylog-server/graylog-certificate.pem
http_tls_key_file = /etc/graylog-server/graylog/graylog-key.pem
http_tls_key_password = secret

[fffhurst (Franklin Hurst)]

I cannot fall back to using http, the certs have to stay in place.

[fffhurst (Franklin Hurst)]

@gsmith

Are you tail’ing your Graylog file when trying to logging the Web interface? if so, what do you see?
Normally if there is a issue with logging into the Web interface this will be shown in Graylog log file.

Nothing at all happens in the graylog log file. It is not possible to get to the graylog web interface to log in. The browser says this site cannot be reached. This same thing happens whether you try to go to the graylog FQDN from another vm on the net or from the graylog server itself. The connection times out.

[fffhurst (Franklin Hurst)]

I think before I start changing those settings it would be good to be able to log into the web interface, correct? I prefer to try to focus on solving one problem at a time.

[gsmith]

Hello,

@fffhurst

The browser says this site cannot be reached. This same thing happens whether you try to go to the graylog FQDN from another vm on the net or from the graylog server itself.

Number one cause of that error would be If the certificates are incorrect and the wrong certificate is inserted in the Keystore or it could be both.

It is possible you can get that same error with the following:

1. Firewall is incorrect or enable.
2. DNS server or /etc/hosts file is incorrect.
3. Selinux/Apparmor is enable or configured incorrect
4. External firewall Cisco/Fortinet could be blocking you FQDN/IP address. ( sometimes ACL’s are in place)
5. Graylog configuration file is Incorrect
6. Permission issues on files and directories ( i.e. Graylog is unable to access files it needs)

Out of curiosity can you PING you Graylog servers ( FDQN) from another node inside this environment?

[fffhurst (Franklin Hurst)]

@gsmith

Out of curiosity can you PING you Graylog servers ( FDQN) from another node inside this environment?

I can ping both the IP and the FQDN from another node inside the environment (as of today).

[gsmith]

Hello,

Sorry for the late response , To be honest not sure, I gave you all the suggestion I know on how to fix this issue. From my over all observation you have a lot of configuration that are incorrect. So basically if there are incorrect configurations your server would run for a few and then crash.

The point is…

@fffhurst

Grayog web admin page is still unavailable.

TOP Things to look at are…

1.Graylog Service
2.Firewall
3.Permission
4.Certificates (Since you stated you can make configuration to HTTPS not sure what to tell you on check those)
5.Configurations and/or Resources