I am pretty sure that is just logging in UTC.
Can you give the graylog serverlog from startup after reboot? As in, reboot, stop the graylog service, set the tail, then start graylog and post the log results? Pre boot logs seem fine.
That makes sense.
Here they are.
sudo tail -f /var/log/graylog-server/server.log
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[?:1.8.0_312]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[?:?]
at org.graylog.shaded.elasticsearch7.org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[?:?]
... 1 more
2021-12-30T09:16:03.085-05:00 ERROR [VersionProbe] Unable to retrieve version from Elasticsearch node: Failed to connect to /127.0.0.1:9200. - Connection refused (Connection refused).
2021-12-30T09:16:04.520-05:00 INFO [InputSetupService] Attempting to close input <org.graylog2.inputs.raw.udp.RawUDPInput.614b86e49f8bf82a3733f849> [Raw/Plaintext UDP].
2021-12-30T09:16:04.540-05:00 INFO [InputSetupService] Input <org.graylog2.inputs.raw.udp.RawUDPInput.614b86e49f8bf82a3733f849> closed. Took [19ms]
2021-12-30T09:18:31.944-05:00 INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2021-12-30T09:18:33.449-05:00 INFO [CmdLineTool] Loaded plugin: AWS plugins 4.2.1 [org.graylog.aws.AWSPlugin]
2021-12-30T09:18:33.453-05:00 INFO [CmdLineTool] Loaded plugin: Integrations 4.2.1 [org.graylog.integrations.IntegrationsPlugin]
2021-12-30T09:18:33.454-05:00 INFO [CmdLineTool] Loaded plugin: Collector 4.2.1 [org.graylog.plugins.collector.CollectorPlugin]
2021-12-30T09:18:33.456-05:00 INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.2.1 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2021-12-30T09:18:33.456-05:00 INFO [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.2.1+5442e44 [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2021-12-30T09:18:33.456-05:00 INFO [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.2.1+5442e44 [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2021-12-30T09:18:33.495-05:00 INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2021-12-30T09:18:33.736-05:00 INFO [Version] HV000001: Hibernate Validator null
2021-12-30T09:18:38.087-05:00 INFO [InputBufferImpl] Message journal is enabled.
2021-12-30T09:18:38.115-05:00 INFO [NodeId] Node ID: 0646dbed-0a28-49e5-bf71-00e9e67fcfd9
2021-12-30T09:18:38.434-05:00 INFO [LogManager] Loading logs.
2021-12-30T09:18:38.474-05:00 WARN [Log] Found a corrupted index file, /var/lib/graylog-server/journal/messagejournal-0/00000000000468958172.index, deleting and rebuilding index...
2021-12-30T09:18:39.789-05:00 INFO [LogManager] Logs loading complete.
2021-12-30T09:18:39.797-05:00 INFO [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2021-12-30T09:18:39.832-05:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2021-12-30T09:18:39.899-05:00 INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2021-12-30T09:18:39.946-05:00 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:12}] to localhost:27017
2021-12-30T09:18:39.952-05:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 27]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=3999732}
2021-12-30T09:18:39.983-05:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:13}] to localhost:27017
2021-12-30T09:18:40.279-05:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2021-12-30T09:18:41.184-05:00 INFO [ElasticsearchVersionProvider] Elasticsearch cluster is running v7.10.2
2021-12-30T09:18:42.369-05:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-12-30T09:18:42.438-05:00 INFO [connection] Opened connection [connectionId{localValue:3, serverValue:14}] to localhost:27017
2021-12-30T09:18:43.035-05:00 INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2021-12-30T09:18:44.502-05:00 INFO [ServerBootstrap] Graylog server 4.2.1+5442e44 starting up
2021-12-30T09:18:44.503-05:00 INFO [ServerBootstrap] JRE: Private Build 1.8.0_312 on Linux 4.15.0-163-generic
2021-12-30T09:18:44.503-05:00 INFO [ServerBootstrap] Deployment: deb
2021-12-30T09:18:44.504-05:00 INFO [ServerBootstrap] OS: Ubuntu 18.04.6 LTS (bionic)
2021-12-30T09:18:44.504-05:00 INFO [ServerBootstrap] Arch: amd64
2021-12-30T09:18:44.573-05:00 INFO [PeriodicalsService] Starting 29 periodicals ...
2021-12-30T09:18:44.573-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2021-12-30T09:18:44.579-05:00 INFO [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2021-12-30T09:18:44.584-05:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2021-12-30T09:18:44.584-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2021-12-30T09:18:44.592-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2021-12-30T09:18:44.598-05:00 INFO [connection] Opened connection [connectionId{localValue:4, serverValue:15}] to localhost:27017
2021-12-30T09:18:44.611-05:00 INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2021-12-30T09:18:44.639-05:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2021-12-30T09:18:44.640-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2021-12-30T09:18:44.642-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2021-12-30T09:18:44.669-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2021-12-30T09:18:44.687-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2021-12-30T09:18:44.682-05:00 INFO [LookupTableService] Data Adapter geoip/614e28029f8bf82a3736d378 [@5701fdad] STARTING
2021-12-30T09:18:44.689-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2021-12-30T09:18:44.700-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2021-12-30T09:18:44.702-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2021-12-30T09:18:44.703-05:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2021-12-30T09:18:44.707-05:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2021-12-30T09:18:44.713-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2021-12-30T09:18:44.720-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2021-12-30T09:18:44.732-05:00 INFO [connection] Opened connection [connectionId{localValue:6, serverValue:16}] to localhost:27017
2021-12-30T09:18:44.755-05:00 INFO [connection] Opened connection [connectionId{localValue:5, serverValue:18}] to localhost:27017
2021-12-30T09:18:44.756-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2021-12-30T09:18:44.763-05:00 INFO [connection] Opened connection [connectionId{localValue:7, serverValue:17}] to localhost:27017
2021-12-30T09:18:45.056-05:00 INFO [LookupDataAdapterRefreshService] Adding job for <geoip/614e28029f8bf82a3736d378/@5701fdad> [interval=60000ms]
2021-12-30T09:18:45.057-05:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2021-12-30T09:18:45.058-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2021-12-30T09:18:45.065-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2021-12-30T09:18:45.069-05:00 INFO [LookupTableService] Data Adapter geoip/614e28029f8bf82a3736d378 [@5701fdad] RUNNING
2021-12-30T09:18:45.077-05:00 INFO [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [3600s].
2021-12-30T09:18:45.077-05:00 INFO [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2021-12-30T09:18:45.077-05:00 INFO [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2021-12-30T09:18:45.078-05:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2021-12-30T09:18:45.078-05:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2021-12-30T09:18:45.079-05:00 INFO [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2021-12-30T09:18:45.085-05:00 INFO [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2021-12-30T09:18:45.085-05:00 INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2021-12-30T09:18:45.294-05:00 INFO [LookupTableService] Cache geoip/614e28469f8bf82a3736d3c2 [@b21af76] STARTING
2021-12-30T09:18:45.301-05:00 INFO [LookupTableService] Cache geoip/614e28469f8bf82a3736d3c2 [@b21af76] RUNNING
2021-12-30T09:18:45.325-05:00 INFO [LookupTableService] Starting lookup table geoip/614e288a9f8bf82a3736d40e [@4598beae] using cache geoip/614e28469f8bf82a3736d3c2 [@b21af76], data adapter geoip/614e28029f8bf82a3736d378 [@5701fdad]
2021-12-30T09:18:51.558-05:00 INFO [NetworkListener] Started listener bound to [172.16.2.15:9000]
2021-12-30T09:18:51.572-05:00 INFO [HttpServer] [HttpServer] Started.
2021-12-30T09:18:51.573-05:00 INFO [JerseyService] Started REST API at <172.16.2.15:9000>
2021-12-30T09:18:51.575-05:00 INFO [ServerBootstrap] Services started, startup times in ms: {FailureHandlingService [RUNNING]=32, PrometheusExporter [RUNNING]=53, JobSchedulerService [RUNNING]=54, UserSessionTerminationService [RUNNING]=54, EtagService [RUNNING]=54, OutputSetupService [RUNNING]=55, BufferSynchronizerService [RUNNING]=58, UrlWhitelistService [RUNNING]=59, LocalKafkaMessageQueueReader [RUNNING]=63, LocalKafkaMessageQueueWriter [RUNNING]=63, GracefulShutdownService [RUNNING]=64, InputSetupService [RUNNING]=66, LocalKafkaJournal [RUNNING]=84, ConfigurationEtagService [RUNNING]=92, MongoDBProcessingStatusRecorderService [RUNNING]=99, StreamCacheService [RUNNING]=540, PeriodicalsService [RUNNING]=555, LookupTableService [RUNNING]=768, JerseyService [RUNNING]=7023}
2021-12-30T09:18:51.584-05:00 INFO [ServiceManagerListener] Services are healthy
2021-12-30T09:18:51.595-05:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2021-12-30T09:18:51.604-05:00 INFO [ServerBootstrap] Graylog server up and running.
2021-12-30T09:18:51.657-05:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/614b86e49f8bf82a3733f849] is now STARTING
2021-12-30T09:18:52.310-05:00 WARN [LookupTableService] Lookup table <geoip-lookup> does not exist
2021-12-30T09:18:52.655-05:00 INFO [InputStateListener] Input [Raw/Plaintext UDP/614b86e49f8bf82a3733f849] is now RUNNING
2021-12-30T09:18:54.340-05:00 INFO [connection] Opened connection [connectionId{localValue:8, serverValue:19}] to localhost:27017
2021-12-30T09:18:54.436-05:00 INFO [connection] Opened connection [connectionId{localValue:9, serverValue:20}] to localhost:27017
sudo sudo tail -f /mnt/elasticsearch/es_log/graylog.log
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.10.2.jar:7.10.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-12-30T09:15:49,360][INFO ][o.e.n.Node ] [graylog] stopped
[2021-12-30T09:15:49,360][INFO ][o.e.n.Node ] [graylog] closing ...
[2021-12-30T09:15:49,431][INFO ][o.e.n.Node ] [graylog] closed
[2021-12-30T09:17:51,402][INFO ][o.e.n.Node ] [graylog] version[7.10.2], pid[2873], build[oss/deb/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.15.0-163-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-12-30T09:17:51,409][INFO ][o.e.n.Node ] [graylog] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-12-30T09:17:51,410][INFO ][o.e.n.Node ] [graylog] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-3903708868125792044, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-12-30T09:17:53,408][INFO ][o.e.p.PluginsService ] [graylog] loaded module [aggs-matrix-stats]
[2021-12-30T09:17:53,409][INFO ][o.e.p.PluginsService ] [graylog] loaded module [analysis-common]
[2021-12-30T09:17:53,410][INFO ][o.e.p.PluginsService ] [graylog] loaded module [geo]
[2021-12-30T09:17:53,410][INFO ][o.e.p.PluginsService ] [graylog] loaded module [ingest-common]
[2021-12-30T09:17:53,411][INFO ][o.e.p.PluginsService ] [graylog] loaded module [ingest-geoip]
[2021-12-30T09:17:53,411][INFO ][o.e.p.PluginsService ] [graylog] loaded module [ingest-user-agent]
[2021-12-30T09:17:53,412][INFO ][o.e.p.PluginsService ] [graylog] loaded module [kibana]
[2021-12-30T09:17:53,412][INFO ][o.e.p.PluginsService ] [graylog] loaded module [lang-expression]
[2021-12-30T09:17:53,413][INFO ][o.e.p.PluginsService ] [graylog] loaded module [lang-mustache]
[2021-12-30T09:17:53,413][INFO ][o.e.p.PluginsService ] [graylog] loaded module [lang-painless]
[2021-12-30T09:17:53,414][INFO ][o.e.p.PluginsService ] [graylog] loaded module [mapper-extras]
[2021-12-30T09:17:53,415][INFO ][o.e.p.PluginsService ] [graylog] loaded module [parent-join]
[2021-12-30T09:17:53,415][INFO ][o.e.p.PluginsService ] [graylog] loaded module [percolator]
[2021-12-30T09:17:53,416][INFO ][o.e.p.PluginsService ] [graylog] loaded module [rank-eval]
[2021-12-30T09:17:53,416][INFO ][o.e.p.PluginsService ] [graylog] loaded module [reindex]
[2021-12-30T09:17:53,417][INFO ][o.e.p.PluginsService ] [graylog] loaded module [repository-url]
[2021-12-30T09:17:53,417][INFO ][o.e.p.PluginsService ] [graylog] loaded module [systemd]
[2021-12-30T09:17:53,418][INFO ][o.e.p.PluginsService ] [graylog] loaded module [transport-netty4]
[2021-12-30T09:17:53,419][INFO ][o.e.p.PluginsService ] [graylog] no plugins loaded
[2021-12-30T09:17:53,473][INFO ][o.e.e.NodeEnvironment ] [graylog] using [1] data paths, mounts [[/mnt/elasticsearch (/dev/sdb)]], net usable_space [406.2gb], net total_space [688gb], types [ext4]
[2021-12-30T09:17:53,474][INFO ][o.e.e.NodeEnvironment ] [graylog] heap size [1gb], compressed ordinary object pointers [true]
[2021-12-30T09:17:53,761][INFO ][o.e.n.Node ] [graylog] node name [graylog], node ID [Lfg5ABAgRtKaa-BepiwdMw], cluster name [graylog], roles [master, remote_cluster_client, data, ingest]
[2021-12-30T09:18:00,891][INFO ][o.e.t.NettyAllocator ] [graylog] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2021-12-30T09:18:01,006][INFO ][o.e.d.DiscoveryModule ] [graylog] using discovery type [single-node] and seed hosts providers [settings]
[2021-12-30T09:18:01,521][WARN ][o.e.g.DanglingIndicesState] [graylog] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-12-30T09:18:01,832][INFO ][o.e.n.Node ] [graylog] initialized
[2021-12-30T09:18:01,833][INFO ][o.e.n.Node ] [graylog] starting ...
[2021-12-30T09:18:02,034][INFO ][o.e.t.TransportService ] [graylog] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2021-12-30T09:18:02,587][INFO ][o.e.c.c.Coordinator ] [graylog] cluster UUID [fdHrTb1WTaaK4sJgTBXS9A]
[2021-12-30T09:18:02,754][INFO ][o.e.c.s.MasterService ] [graylog] elected-as-master ([1] nodes joined)[{graylog}{Lfg5ABAgRtKaa-BepiwdMw}{41ZXWFjBR9mICFcu3_ea7g}{127.0.0.1}{127.0.0.1:9300}{dimr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 12, version: 1257, delta: master node changed {previous [], current [{graylog}{Lfg5ABAgRtKaa-BepiwdMw}{41ZXWFjBR9mICFcu3_ea7g}{127.0.0.1}{127.0.0.1:9300}{dimr}]}
[2021-12-30T09:18:02,929][INFO ][o.e.c.s.ClusterApplierService] [graylog] master node changed {previous [], current [{graylog}{Lfg5ABAgRtKaa-BepiwdMw}{41ZXWFjBR9mICFcu3_ea7g}{127.0.0.1}{127.0.0.1:9300}{dimr}]}, term: 12, version: 1257, reason: Publication{term=12, version=1257}
[2021-12-30T09:18:02,958][INFO ][o.e.h.AbstractHttpServerTransport] [graylog] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2021-12-30T09:18:02,959][INFO ][o.e.n.Node ] [graylog] started
[2021-12-30T09:18:03,406][INFO ][o.e.g.GatewayService ] [graylog] recovered [28] indices into cluster_state
[2021-12-30T09:18:16,829][INFO ][o.e.c.r.a.AllocationService] [graylog] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[gl-events_0][0]]]).
tail -f /var/log/elasticsearch/gc.log
[2021-12-30T14:15:47.649+0000][1307][safepoint ] Safepoint "Cleanup", Time since last: 1020793783 ns, Reaching safepoint: 8085550 ns, At safepoint: 62601 ns, Total: 8148151 ns
[2021-12-30T14:15:48.653+0000][1307][safepoint ] Safepoint "Cleanup", Time since last: 1000163980 ns, Reaching safepoint: 4534528 ns, At safepoint: 43700 ns, Total: 4578228 ns
[2021-12-30T14:15:49.669+0000][1307][safepoint ] Safepoint "Cleanup", Time since last: 1003014934 ns, Reaching safepoint: 12759876 ns, At safepoint: 37300 ns, Total: 12797176 ns
[2021-12-30T14:15:50.676+0000][1307][safepoint ] Safepoint "Cleanup", Time since last: 1000309857 ns, Reaching safepoint: 6983541 ns, At safepoint: 8900 ns, Total: 6992441 ns
[2021-12-30T14:15:51.677+0000][1307][safepoint ] Safepoint "Cleanup", Time since last: 1000132103 ns, Reaching safepoint: 187401 ns, At safepoint: 20000 ns, Total: 207401 ns
[2021-12-30T14:15:52.029+0000][1307][gc,heap,exit] Heap
[2021-12-30T14:15:52.029+0000][1307][gc,heap,exit] garbage-first heap total 1048576K, used 527442K [0x00000000c0000000, 0x0000000100000000)
[2021-12-30T14:15:52.029+0000][1307][gc,heap,exit] region size 1024K, 404 young (413696K), 17 survivors (17408K)
[2021-12-30T14:15:52.029+0000][1307][gc,heap,exit] Metaspace used 74117K, capacity 75804K, committed 76700K, reserved 1116160K
[2021-12-30T14:15:52.029+0000][1307][gc,heap,exit] class space used 9102K, capacity 9635K, committed 9708K, reserved 1048576K
tail -f /var/log/elasticsearch/gc.log
[2021-12-30T14:19:26.025+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1007718983 ns, Reaching safepoint: 267601 ns, At safepoint: 26001 ns, Total: 293602 ns
[2021-12-30T14:19:27.026+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000112003 ns, Reaching safepoint: 283102 ns, At safepoint: 25500 ns, Total: 308602 ns
[2021-12-30T14:19:28.028+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000178064 ns, Reaching safepoint: 2316712 ns, At safepoint: 22900 ns, Total: 2339612 ns
[2021-12-30T14:19:29.029+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000111525 ns, Reaching safepoint: 276901 ns, At safepoint: 84600 ns, Total: 361501 ns
[2021-12-30T14:19:30.029+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000141987 ns, Reaching safepoint: 304201 ns, At safepoint: 19900 ns, Total: 324101 ns
[2021-12-30T14:19:31.033+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1003414065 ns, Reaching safepoint: 167501 ns, At safepoint: 16300 ns, Total: 183801 ns
[2021-12-30T14:19:32.033+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000150011 ns, Reaching safepoint: 144601 ns, At safepoint: 18700 ns, Total: 163301 ns
[2021-12-30T14:19:34.034+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 2000296112 ns, Reaching safepoint: 213901 ns, At safepoint: 20701 ns, Total: 234602 ns
[2021-12-30T14:19:35.034+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000173302 ns, Reaching safepoint: 186201 ns, At safepoint: 24800 ns, Total: 211001 ns
[2021-12-30T14:19:36.034+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000157768 ns, Reaching safepoint: 200201 ns, At safepoint: 6800 ns, Total: 207001 ns
[2021-12-30T14:19:37.035+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000191133 ns, Reaching safepoint: 219701 ns, At safepoint: 21500 ns, Total: 241201 ns
[2021-12-30T14:19:39.035+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 2000315464 ns, Reaching safepoint: 223501 ns, At safepoint: 8800 ns, Total: 232301 ns
[2021-12-30T14:19:40.036+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000456235 ns, Reaching safepoint: 181500 ns, At safepoint: 22800 ns, Total: 204300 ns
[2021-12-30T14:19:41.037+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000451202 ns, Reaching safepoint: 151400 ns, At safepoint: 27001 ns, Total: 178401 ns
[2021-12-30T14:19:43.037+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 2000458705 ns, Reaching safepoint: 219201 ns, At safepoint: 8500 ns, Total: 227701 ns
[2021-12-30T14:19:44.038+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000419007 ns, Reaching safepoint: 288202 ns, At safepoint: 7700 ns, Total: 295902 ns
[2021-12-30T14:19:45.039+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000389976 ns, Reaching safepoint: 310101 ns, At safepoint: 5900 ns, Total: 316001 ns
[2021-12-30T14:19:46.040+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000556948 ns, Reaching safepoint: 175701 ns, At safepoint: 20500 ns, Total: 196201 ns
[2021-12-30T14:19:47.040+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000174116 ns, Reaching safepoint: 153001 ns, At safepoint: 7700 ns, Total: 160701 ns
[2021-12-30T14:19:48.040+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000171787 ns, Reaching safepoint: 170201 ns, At safepoint: 7200 ns, Total: 177401 ns
[2021-12-30T14:19:54.041+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 6000758837 ns, Reaching safepoint: 146000 ns, At safepoint: 27600 ns, Total: 173600 ns
[2021-12-30T14:19:55.042+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000209997 ns, Reaching safepoint: 230201 ns, At safepoint: 5600 ns, Total: 235801 ns
[2021-12-30T14:19:56.042+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000648273 ns, Reaching safepoint: 158700 ns, At safepoint: 7300 ns, Total: 166000 ns
[2021-12-30T14:19:57.043+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000194646 ns, Reaching safepoint: 211301 ns, At safepoint: 7200 ns, Total: 218501 ns
[2021-12-30T14:20:02.044+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 5001030763 ns, Reaching safepoint: 184401 ns, At safepoint: 23700 ns, Total: 208101 ns
[2021-12-30T14:20:04.045+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 2000340682 ns, Reaching safepoint: 180801 ns, At safepoint: 23800 ns, Total: 204601 ns
[2021-12-30T14:20:05.045+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000184357 ns, Reaching safepoint: 214601 ns, At safepoint: 23200 ns, Total: 237801 ns
[2021-12-30T14:20:06.045+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000186236 ns, Reaching safepoint: 172400 ns, At safepoint: 9000 ns, Total: 181400 ns
[2021-12-30T14:20:07.046+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000162514 ns, Reaching safepoint: 150101 ns, At safepoint: 23600 ns, Total: 173701 ns
[2021-12-30T14:20:08.046+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000191393 ns, Reaching safepoint: 161101 ns, At safepoint: 9000 ns, Total: 170101 ns
[2021-12-30T14:20:10.047+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 2000486824 ns, Reaching safepoint: 175401 ns, At safepoint: 21300 ns, Total: 196701 ns
[2021-12-30T14:20:14.856+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 4808851041 ns, Reaching safepoint: 361402 ns, At safepoint: 12800 ns, Total: 374202 ns
[2021-12-30T14:20:14.963+0000][2873][safepoint ] Safepoint "ICBufferFull", Time since last: 106574300 ns, Reaching safepoint: 427501 ns, At safepoint: 12100 ns, Total: 439601 ns
[2021-12-30T14:20:15.963+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000217337 ns, Reaching safepoint: 216901 ns, At safepoint: 21100 ns, Total: 238001 ns
[2021-12-30T14:20:16.964+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000178719 ns, Reaching safepoint: 162800 ns, At safepoint: 33200 ns, Total: 196000 ns
[2021-12-30T14:20:17.964+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000208001 ns, Reaching safepoint: 161900 ns, At safepoint: 11301 ns, Total: 173201 ns
[2021-12-30T14:20:21.965+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 4000520124 ns, Reaching safepoint: 158501 ns, At safepoint: 8800 ns, Total: 167301 ns
[2021-12-30T14:20:24.966+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 3000642392 ns, Reaching safepoint: 214801 ns, At safepoint: 19300 ns, Total: 234101 ns
[2021-12-30T14:20:27.966+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 3000470848 ns, Reaching safepoint: 138401 ns, At safepoint: 6000 ns, Total: 144401 ns
[2021-12-30T14:20:30.967+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 3000448210 ns, Reaching safepoint: 155201 ns, At safepoint: 7300 ns, Total: 162501 ns
[2021-12-30T14:20:31.967+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000169074 ns, Reaching safepoint: 189801 ns, At safepoint: 4600 ns, Total: 194401 ns
[2021-12-30T14:20:33.968+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 2000548607 ns, Reaching safepoint: 161600 ns, At safepoint: 20500 ns, Total: 182100 ns
[2021-12-30T14:20:34.969+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000165332 ns, Reaching safepoint: 233501 ns, At safepoint: 24600 ns, Total: 258101 ns
[2021-12-30T14:20:35.969+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000196318 ns, Reaching safepoint: 184601 ns, At safepoint: 29800 ns, Total: 214401 ns
[2021-12-30T14:20:36.969+0000][2873][safepoint ] Safepoint "Cleanup", Time since last: 1000181405 ns, Reaching safepoint: 147501 ns, At safepoint: 23900 ns, Total: 171401 ns
Lets focus on the raw input for fortigate. We should see something there. double check all simple things, make sure the port is matching the expected port of data from fortigate. You can watch the port to make sure it sees traffic coming in (there are commands in the community you can research on how to do that… (busy day here)) You could even rebuild the port in case there is an issue there?
Point being that even if Elasticsearch was down, you should still see Graylog receiving and queuing up data for Elastic. Maybe a queued OS configuration change before the reboot, maybe OS security or related that is blocking the port?
Happy New Year.
Can you give me a hit on how to do this?
I rebooted the server, no messages. Ran netstat -tu and tcpdump port 514. I don’t particuarly understand the the output.
netstat -tu
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54436 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53546 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:52822 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54412 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53450 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54148 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53802 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:55186 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53616 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53962 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53914 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:52062 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54116 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53562 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53426 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53228 TIME_WAIT
tcp 0 0 localhost:27017 localhost:33182 ESTABLISHED
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54210 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:52128 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54634 TIME_WAIT
tcp 0 0 localhost:27017 localhost:33180 ESTABLISHED
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53444 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54484 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54554 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:52094 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53682 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53032 TIME_WAIT
tcp 0 0 localhost:27017 localhost:33202 ESTABLISHED
tcp 0 0 graylog:zabbix-agent 192.168.1.1:52772 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53476 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53306 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53162 TIME_WAIT
tcp 0 0 localhost:27017 localhost:33184 ESTABLISHED
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54196 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53752 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53622 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:52202 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:52942 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53684 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53678 TIME_WAIT
tcp 0 0 localhost:27017 localhost:33186 ESTABLISHED
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54456 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54292 TIME_WAIT
tcp 0 0 localhost:27017 localhost:33172 ESTABLISHED
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53360 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54542 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54344 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53784 TIME_WAIT
tcp 0 1408 graylog:ssh 192.168.1.3:51362 ESTABLISHED
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53758 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54700 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53512 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53354 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53930 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53482 TIME_WAIT
tcp 0 0 localhost:27017 localhost:33198 ESTABLISHED
tcp 0 0 localhost:27017 localhost:33174 ESTABLISHED
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54022 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54736 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:54018 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53296 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53106 TIME_WAIT
tcp 0 0 localhost:27017 localhost:33200 ESTABLISHED
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53952 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53872 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53796 TIME_WAIT
tcp 0 0 graylog:zabbix-agent 192.168.1.1:53550 TIME_WAIT
tcp6 0 0 graylog:9000 graylog:40542 FIN_WAIT2
tcp6 0 0 localhost:33198 localhost:27017 ESTABLISHED
tcp6 0 0 graylog:9000 192.168.1.2:65157 TIME_WAIT
tcp6 0 0 localhost:33202 localhost:27017 ESTABLISHED
tcp6 0 0 localhost:9200 localhost:33310 ESTABLISHED
tcp6 0 0 localhost:9200 localhost:33312 ESTABLISHED
tcp6 0 0 localhost:33186 localhost:27017 ESTABLISHED
tcp6 0 0 graylog:9000 192.168.1.2:65149 ESTABLISHED
tcp6 0 0 graylog:9000 192.168.1.2:65156 TIME_WAIT
tcp6 0 0 localhost:33174 localhost:27017 ESTABLISHED
tcp6 0 0 localhost:33310 localhost:9200 ESTABLISHED
tcp6 0 0 localhost:9200 localhost:33296 ESTABLISHED
tcp6 0 0 graylog:9000 192.168.1.3:51440 TIME_WAIT
tcp6 0 0 localhost:33306 localhost:9200 ESTABLISHED
tcp6 0 0 localhost:9200 localhost:33308 ESTABLISHED
tcp6 0 0 graylog:9000 192.168.1.2:65162 TIME_WAIT
tcp6 0 0 localhost:33312 localhost:9200 ESTABLISHED
tcp6 0 0 localhost:33296 localhost:9200 ESTABLISHED
tcp6 0 0 graylog:9000 192.168.1.2:65158 ESTABLISHED
tcp6 0 0 graylog:9000 graylog:40540 ESTABLISHED
tcp6 0 0 localhost:33182 localhost:27017 ESTABLISHED
tcp6 0 0 localhost:9200 localhost:33306 ESTABLISHED
tcp6 0 0 localhost:9200 localhost:33314 ESTABLISHED
tcp6 0 0 localhost:33314 localhost:9200 ESTABLISHED
tcp6 0 0 graylog:9000 192.168.1.3:51438 ESTABLISHED
tcp6 0 0 localhost:33200 localhost:27017 ESTABLISHED
tcp6 0 0 localhost:33184 localhost:27017 ESTABLISHED
tcp6 0 0 graylog:9000 192.168.1.2:65151 TIME_WAIT
tcp6 0 0 graylog:9000 192.168.1.3:51442 ESTABLISHED
tcp6 0 0 localhost:33172 localhost:27017 ESTABLISHED
tcp6 0 0 graylog:9000 192.168.1.3:51443 ESTABLISHED
tcp6 0 0 localhost:33308 localhost:9200 ESTABLISHED
tcp6 0 0 graylog:40540 graylog:9000 ESTABLISHED
tcp6 0 0 graylog:9000 192.168.1.2:65155 TIME_WAIT
tcp6 1 0 graylog:40542 graylog:9000 CLOSE_WAIT
tcp6 0 0 graylog:9000 192.168.1.3:51441 TIME_WAIT
tcp6 0 0 graylog:9000 192.168.1.2:65152 ESTABLISHED
tcp6 0 0 localhost:33180 localhost:27017 ESTABLISHED
tcp6 0 0 graylog:9000 192.168.1.3:51439 TIME_WAIT
tcpdump port 514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
10:47:51.139794 IP _gateway.11421 > graylog.syslog: SYSLOG local7.notice, length: 586
10:47:51.193657 IP _gateway.6904 > graylog.syslog: SYSLOG local7.info, length: 632
10:47:51.229805 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 1001
10:47:51.229805 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 781
10:47:51.229850 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 1002
10:47:51.239739 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 1018
10:47:51.239739 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 1018
10:47:51.278825 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 924
10:47:51.309783 IP _gateway.11421 > graylog.syslog: SYSLOG local7.notice, length: 998
10:47:51.349825 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 883
10:47:51.359772 IP _gateway.11421 > graylog.syslog: SYSLOG local7.notice, length: 909
10:47:51.359772 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 909
10:47:51.359792 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 965
10:47:51.359792 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 682
10:47:51.359799 IP _gateway.20703 > graylog.syslog: SYSLOG local7.notice, length: 908
10:47:51.359799 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 894
10:47:51.359805 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 966
10:47:51.369808 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 894
10:47:51.369808 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 864
10:47:52.003607 IP _gateway.11421 > graylog.syslog: SYSLOG local7.notice, length: 1018
10:47:52.015388 IP _gateway.11421 > graylog.syslog: SYSLOG local7.warning, length: 890
10:47:52.019995 IP _gateway.11696 > graylog.syslog: SYSLOG local7.warning, length: 843
10:47:52.020033 IP _gateway.11696 > graylog.syslog: SYSLOG local7.warning, length: 840
10:47:52.030159 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 907
10:47:52.030159 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 906
10:47:52.049996 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 966
10:47:52.050037 IP _gateway.11421 > graylog.syslog: SYSLOG local7.notice, length: 905
10:47:52.058761 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 909
10:47:52.059769 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 998
10:47:52.059769 IP _gateway.11421 > graylog.syslog: SYSLOG local7.notice, length: 883
10:47:52.059808 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 965
10:47:52.059815 IP _gateway.20703 > graylog.syslog: SYSLOG local7.notice, length: 905
10:47:52.059816 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 907
10:47:52.059821 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 906
10:47:52.059830 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 908
10:47:52.069808 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 965
10:47:52.079752 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 907
10:47:52.119830 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 589
10:47:52.122688 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 1027
10:47:52.138544 IP 192.168.1.4.4334 > graylog.syslog: SYSLOG local7.notice, length: 658
10:47:52.138642 IP 192.168.1.4.4334 > graylog.syslog: SYSLOG local7.notice, length: 659
10:47:52.138659 IP 192.168.1.4.4334 > graylog.syslog: SYSLOG local7.notice, length: 652
10:47:52.138665 IP 192.168.1.4.4334 > graylog.syslog: SYSLOG local7.notice, length: 653
10:47:52.169837 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 924
10:47:52.219982 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 971
10:47:52.220024 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 588
10:47:52.245507 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 908
10:47:52.245507 IP _gateway.20703 > graylog.syslog: SYSLOG local7.notice, length: 953
10:47:52.245547 IP _gateway.20703 > graylog.syslog: SYSLOG local7.notice, length: 894
10:47:52.245547 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 902
10:47:52.248822 IP _gateway.11421 > graylog.syslog: SYSLOG local7.notice, length: 910
10:47:52.249771 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 916
10:47:52.249771 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 965
10:47:52.249790 IP _gateway.11421 > graylog.syslog: SYSLOG local7.notice, length: 894
10:47:52.259831 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 1001
10:47:52.259831 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 1002
10:47:52.259865 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 894
10:47:52.259875 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 905
10:47:52.269851 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 894
10:47:52.269851 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 907
10:47:52.289822 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 905
10:47:52.289822 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 905
10:47:52.289848 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 902
10:47:52.289848 IP _gateway.11421 > graylog.syslog: SYSLOG local7.notice, length: 905
10:47:52.289854 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 952
10:47:52.299843 IP _gateway.20703 > graylog.syslog: SYSLOG local7.notice, length: 907
10:47:52.350639 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 998
10:47:52.398866 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 914
10:47:52.399854 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 924
10:47:52.438825 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 970
10:47:52.498863 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 579
10:47:52.498863 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 579
10:47:52.519845 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 920
10:47:52.519845 IP _gateway.20703 > graylog.syslog: SYSLOG local7.notice, length: 599
10:47:52.569833 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 998
10:47:52.589791 IP _gateway.11421 > graylog.syslog: SYSLOG local7.notice, length: 1018
10:47:52.649873 IP _gateway.11421 > graylog.syslog: SYSLOG local7.warning, length: 633
10:47:52.769917 IP _gateway.20703 > graylog.syslog: SYSLOG local7.notice, length: 925
10:47:52.799864 IP _gateway.11696 > graylog.syslog: SYSLOG local7.warning, length: 685
10:47:52.799864 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 961
10:47:52.829859 IP _gateway.20703 > graylog.syslog: SYSLOG local7.notice, length: 998
10:47:52.829900 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 980
10:47:52.852686 IP _gateway.20703 > graylog.syslog: SYSLOG local7.notice, length: 923
10:47:52.852729 IP _gateway.20703 > graylog.syslog: SYSLOG local7.notice, length: 998
10:47:52.870045 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 1018
10:47:52.909917 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 971
10:47:52.979853 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 600
10:47:53.013652 IP _gateway.20703 > graylog.syslog: SYSLOG local7.info, length: 364
10:47:53.029907 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 971
10:47:53.029907 IP _gateway.20703 > graylog.syslog: SYSLOG local7.notice, length: 598
10:47:53.059846 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 924
10:47:53.069901 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 980
10:47:53.099877 IP _gateway.11421 > graylog.syslog: SYSLOG local7.notice, length: 998
10:47:53.099920 IP _gateway.11421 > graylog.syslog: SYSLOG local7.notice, length: 921
10:47:53.128929 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 599
10:47:53.198903 IP _gateway.20703 > graylog.syslog: SYSLOG local7.notice, length: 682
10:47:53.269892 IP _gateway.20374 > graylog.syslog: SYSLOG local7.warning, length: 843
10:47:53.279853 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 998
10:47:53.279909 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 1018
10:47:53.289925 IP _gateway.11421 > graylog.syslog: SYSLOG local7.warning, length: 897
10:47:53.289981 IP _gateway.24094 > graylog.syslog: SYSLOG local7.warning, length: 687
10:47:53.298898 IP _gateway.20703 > graylog.syslog: SYSLOG local7.warning, length: 843
10:47:53.359890 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 998
10:47:53.409946 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 905
10:47:53.410011 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 965
10:47:53.410021 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 902
10:47:58.770117 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 589
10:47:58.770135 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 588
10:47:58.770140 IP _gateway.6904 > graylog.syslog: SYSLOG local7.notice, length: 1049
10:47:58.814964 IP _gateway.20703 > graylog.syslog: SYSLOG local7.notice, length: 918
10:47:58.819919 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 918
10:47:58.830246 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 918
10:47:58.830982 IP _gateway.11421 > graylog.syslog: SYSLOG local7.alert, length: 727
10:47:58.839906 IP _gateway.11421 > graylog.syslog: SYSLOG local7.notice, length: 918
10:47:58.839906 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 998
10:47:58.839956 IP _gateway.11421 > graylog.syslog: SYSLOG local7.notice, length: 918
10:47:58.839956 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 918
10:47:58.869880 IP _gateway.11696 > graylog.syslog: SYSLOG local7.notice, length: 918
10:47:58.900089 IP _gateway.24094 > graylog.syslog: SYSLOG local7.notice, length: 925
10:47:58.968995 IP _gateway.20703 > graylog.syslog: SYSLOG local7.notice, length: 925
10:47:59.019899 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 579
10:47:59.026296 IP _gateway.20703 > graylog.syslog: SYSLOG local7.info, length: 364
10:47:59.089971 IP _gateway.20374 > graylog.syslog: SYSLOG local7.notice, length: 990
^C
521 packets captured
543 packets received by filter
22 packets dropped by kernel
I guess I am surpriced to find traffic on port 514. Is that the Fortigate or is the Graylog server listening to itself? Does this indicate there is a problem getting the traffic form 514 to 1514?
in previous posts you have the fortigate receiving on 1415 - assumes that FortiGate sends to Graylog on port 1415 or you are using something like iptables to route from whatever FortiGate is sending to (514 maybe?) over to port 1415 before Graylog receives it. Graylog shouldn’t be receiving on 514 since it is a protected port… unless you have messed around to allow that… which is possible too, though unadvised. regardless, make sure the port sent from FortiGate is aligned either directly to 1415 or whatever you have in between to make it go to 1415 is set up correctly. For instance it might make sense that an iptables change or misconfiguration doesn’t take place until reboot.
1 Like
That is correct. In the FortiGate UI I cannot change the port number. So I used iptables to send the traffic to 1514.
Would you expect to see the traffic on 1514 instead of 514?
I followed the instruction in the Graylog docs. So I don’t think so.
So this ended up being the problem. When I was rebooting something happened to the iptables. Though I’m not sure what, you can see them in my first post.
I think (it was yesterday afternoon) I ran this command to make it work.
sudo iptables-save > /etc/iptables/rules.v4
ldog@graylog:~$ cat /etc/iptables/rules.v4
# Generated by iptables-save v1.6.1 on Mon Jan 3 15:58:09 2022
*filter
:INPUT ACCEPT [976058976:1468582239457]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [573626405:1124861796006]
COMMIT
# Completed on Mon Jan 3 15:58:09 2022
# Generated by iptables-save v1.6.1 on Mon Jan 3 15:58:09 2022
*nat
:PREROUTING ACCEPT [404:39893]
:INPUT ACCEPT [71:5113]
:OUTPUT ACCEPT [68:5074]
:POSTROUTING ACCEPT [68:5074]
-A PREROUTING -p tcp -m tcp --dport 514 -j REDIRECT --to-ports 1514
-A PREROUTING -p tcp -m tcp --dport 514 -j REDIRECT --to-ports 1514
-A PREROUTING -p udp -m udp --dport 514 -j REDIRECT --to-ports 1514
-A PREROUTING -p udp -m udp --dport 514 -j REDIRECT --to-ports 1514
COMMIT
# Completed on Mon Jan 3 15:58:09 2022
Now I’m trying to understand iptables (which seems complicated at best). Shouldn’t those rules I added be under :PREROUTING ACCEPT [404:39893]?
It also seems like I should delete the duplicate rules. But the instructions I was finding online seemed like it might delete both lines, since the lines are exactly the same.
I was also hesitant to play around with it since it’s finally working!
What? Its working? You mean all of it!??! 
I don’t know enough about IPTABLES to help with configuration… I say let it run and be happy for now… spend some time on researching iptables - there are lots of posts on it in the community…
That was it.
Thanks @tmacgbay and @gsmith for all your help, I really appropriate it. I definitely learned some stuff along the way.
2 Likes
Glad you fixed your issue 
BTW I seen this in the forum.
1 Like
@gsmith found my post on the iptables subject. Thanks!
To summarize:
-
When you make an iptables change it will not persist across reboots. To save the config I use:
sudo /sbin/iptables-save -
When you make a change to iptables it also doesn’t effect existing flows. If your Fortinet box is sending udp packets to 514 it will just stick there and the redirect to 1514 will never happen. Once you realize that’s what’s happening you can reboot the box and that will clear the iptables state. But I found a much easier way.
# If conntrack is not already installed...
sudo yum update
sudo yum install conntrack
# List all existing connections...
sudo conntrack -L
# Flush all connections in the state table...
sudo conntrack -F
The conntrack utility does the flush. After the flush you’ll see the redirect from 514 to 1514 starts immediately (along with any other changes you’ve made to iptables).
Sometimes I wish Linux people were better at documenting how things work! It took me about six months from finding the connection tracking problem to finding a solution. (Don’t worry, it’s not my full time job. It wasn’t six months of full-time effort!)
Hope that helps.
3 Likes
Thanks @danmassa7. I also read through your post that @gsmith mentioned and found that helpful.