Graylog cluster : high cpu on a node even if there's no active input nor processing

sdu · September 27, 2018, 7:46am

Hi,
I noticed a strange behavior when adding a node on my graylog cluster :
graylog-server is running on this new node, which is well integrated within the other nodes of the cluster; the udp load balancer isn’t yet sending messages to it. Moreover, I disabled all inputs on this new node. But graylog-server process is still hitting near 100% of its cpu cores.

here’s my config :

1 nginx udp load balancer for syslog messages
4 graylog latest version nodes
1 elascticsearch 5.6 + mongodb server 3.4

Thanx in advance for any clue.

jan · September 27, 2018, 8:11am

you might find the reason in the Graylog server.log of the node that is going wild.

sdu · September 27, 2018, 12:42pm

log seems fine since last start :

2018-09-27T09:36:40.865+02:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 2.4.6 [org.graylog.aws.plugin.AWSPlugin]
2018-09-27T09:36:40.875+02:00 INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.4.6 [org.graylog.plugins.beats.BeatsInputPlugin]
2018-09-27T09:36:40.876+02:00 INFO  [CmdLineTool] Loaded plugin: CEF Input 2.4.6 [org.graylog.plugins.cef.CEFInputPlugin]
2018-09-27T09:36:40.877+02:00 INFO  [CmdLineTool] Loaded plugin: Collector 2.4.6 [org.graylog.plugins.collector.CollectorPlugin]
2018-09-27T09:36:40.878+02:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.4.6 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2018-09-27T09:36:40.879+02:00 INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.4.6 [org.graylog.plugins.map.MapWidgetPlugin]
2018-09-27T09:36:40.880+02:00 INFO  [CmdLineTool] Loaded plugin: NetFlow Plugin 2.4.6 [org.graylog.plugins.netflow.NetFlowPlugin]
2018-09-27T09:36:40.888+02:00 INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.4.6 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2018-09-27T09:36:40.888+02:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 2.4.6 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2018-09-27T09:36:41.558+02:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms2g -Xmx2g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2018-09-27T09:36:41.848+02:00 INFO  [Version] HV000001: Hibernate Validator 5.1.3.Final
2018-09-27T09:36:44.699+02:00 INFO  [InputBufferImpl] Message journal is enabled.
2018-09-27T09:36:44.743+02:00 INFO  [NodeId] Node ID: 4b3eab03-dc2c-4167-a9a2-3013aa8ea98f
2018-09-27T09:36:44.962+02:00 INFO  [LogManager] Loading logs.
2018-09-27T09:36:45.059+02:00 INFO  [LogManager] Logs loading complete.
2018-09-27T09:36:45.059+02:00 INFO  [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2018-09-27T09:36:45.080+02:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <YieldingWaitStrategy>, running 2 parallel message handlers.
2018-09-27T09:36:45.104+02:00 INFO  [cluster] Cluster created with settings {hosts=[sonde-bdd:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2018-09-27T09:36:45.155+02:00 INFO  [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=sonde-bdd:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2018-09-27T09:36:45.195+02:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:171}] to sonde-bdd:27017
2018-09-27T09:36:45.202+02:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=sonde-bdd:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 4, 17]}, minWireVersion=0, maxWireVersion=5, maxDocumentSize=16777216, roundTripTimeNanos=763557}
2018-09-27T09:36:45.210+02:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:172}] to sonde-bdd:27017
2018-09-27T09:36:45.641+02:00 INFO  [AbstractJestClient] Setting server pool to a list of 1 servers: [http://10.4.1.57:9200]
2018-09-27T09:36:45.642+02:00 INFO  [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2018-09-27T09:36:45.734+02:00 INFO  [JestClientFactory] Using custom ObjectMapper instance
2018-09-27T09:36:45.734+02:00 INFO  [JestClientFactory] Node Discovery disabled...
2018-09-27T09:36:45.734+02:00 INFO  [JestClientFactory] Idle connection reaping disabled...
2018-09-27T09:36:46.011+02:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <YieldingWaitStrategy>.
2018-09-27T09:36:48.388+02:00 INFO  [RulesEngineProvider] No static rules file loaded.
2018-09-27T09:36:48.822+02:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /var/local/GeoLite2-City.mmdb
2018-09-27T09:36:48.834+02:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:173}] to sonde-bdd:27017
2018-09-27T09:36:48.847+02:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <YieldingWaitStrategy>.
2018-09-27T09:36:49.196+02:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /var/local/GeoLite2-City.mmdb
2018-09-27T09:36:49.395+02:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /var/local/GeoLite2-City.mmdb
2018-09-27T09:36:49.543+02:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /var/local/GeoLite2-City.mmdb
2018-09-27T09:36:49.716+02:00 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /var/local/GeoLite2-City.mmdb
2018-09-27T09:36:50.068+02:00 INFO  [ServerBootstrap] Graylog server 2.4.6+ceaa7e4 starting up
2018-09-27T09:36:50.068+02:00 INFO  [ServerBootstrap] JRE: Oracle Corporation 1.8.0_181 on Linux 4.15.0-20-generic
2018-09-27T09:36:50.068+02:00 INFO  [ServerBootstrap] Deployment: deb
2018-09-27T09:36:50.069+02:00 INFO  [ServerBootstrap] OS: Ubuntu 18.04.1 LTS (bionic)
2018-09-27T09:36:50.069+02:00 INFO  [ServerBootstrap] Arch: amd64
2018-09-27T09:36:50.109+02:00 INFO  [PeriodicalsService] Starting 25 periodicals ...
2018-09-27T09:36:50.110+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2018-09-27T09:36:50.116+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.116+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2018-09-27T09:36:50.126+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.127+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.128+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2018-09-27T09:36:50.130+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.132+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.IndexRetentionThread] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.132+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.IndexRotationThread] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.134+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2018-09-27T09:36:50.135+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.VersionCheckThread] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.135+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2018-09-27T09:36:50.137+02:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2018-09-27T09:36:50.140+02:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:174}] to sonde-bdd:27017
2018-09-27T09:36:50.142+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.143+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.145+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.150+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.150+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.150+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.150+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.150+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.156+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2018-09-27T09:36:50.159+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2018-09-27T09:36:50.161+02:00 INFO  [PeriodicalsService] Not starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.161+02:00 INFO  [PeriodicalsService] Not starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical. Only started on Graylog master nodes.
2018-09-27T09:36:50.188+02:00 INFO  [LookupTableService] Data Adapter otx-api-ip/5a53280a3656ae5184db21ff [@530ac3] STARTING
2018-09-27T09:36:50.190+02:00 WARN  [OTXDataAdapter] OTX API key is missing. Make sure to add the key to allow higher request limits.
2018-09-27T09:36:50.195+02:00 INFO  [LookupTableService] Data Adapter abuse-ch-ransomware-domains/5a53280a3656ae5184db21fd [@1464d747] STARTING
2018-09-27T09:36:50.207+02:00 INFO  [LookupTableService] Data Adapter whois/5a53280a3656ae5184db2202 [@ac8a9] STARTING
2018-09-27T09:36:50.209+02:00 INFO  [LookupTableService] Data Adapter whois/5a53280a3656ae5184db2202 [@ac8a9] RUNNING
2018-09-27T09:36:50.196+02:00 ERROR [LookupDataAdapter] Couldn't start data adapter <abuse-ch-ransomware-domains/5a53280a3656ae5184db21fd/@1464d747>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Abuse.ch service is disabled, not starting adapter. To enable it please go to System / Configurations.
	at org.graylog.plugins.threatintel.adapters.abusech.AbuseChRansomAdapter.doStart(AbuseChRansomAdapter.java:80) ~[?:?]
	at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
	at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
	at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2018-09-27T09:36:50.215+02:00 INFO  [LookupTableService] Data Adapter abuse-ch-ransomware-domains/5a53280a3656ae5184db21fd [@1464d747] RUNNING
2018-09-27T09:36:50.216+02:00 INFO  [LookupTableService] Data Adapter tor-exit-node/5a53280a3656ae5184db21fe [@21c4e6fc] STARTING
2018-09-27T09:36:50.218+02:00 INFO  [LookupTableService] Data Adapter spamhaus-drop/5a53280a3656ae5184db2200 [@3fccbf84] STARTING
2018-09-27T09:36:50.219+02:00 INFO  [LookupTableService] Data Adapter otx-api-domain/5a53280a3656ae5184db2201 [@27b70cec] STARTING
2018-09-27T09:36:50.219+02:00 WARN  [OTXDataAdapter] OTX API key is missing. Make sure to add the key to allow higher request limits.
2018-09-27T09:36:50.217+02:00 ERROR [LookupDataAdapter] Couldn't start data adapter <tor-exit-node/5a53280a3656ae5184db21fe/@21c4e6fc>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: TOR service is disabled, not starting TOR exit addresses adapter. To enable it please go to System / Configurations.
	at org.graylog.plugins.threatintel.adapters.tor.TorExitNodeDataAdapter.doStart(TorExitNodeDataAdapter.java:73) ~[?:?]
	at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
	at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
	at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2018-09-27T09:36:50.219+02:00 ERROR [LookupDataAdapter] Couldn't start data adapter <spamhaus-drop/5a53280a3656ae5184db2200/@3fccbf84>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Spamhaus service is disabled, not starting (E)DROP adapter. To enable it please go to System / Configurations.
	at org.graylog.plugins.threatintel.adapters.spamhaus.SpamhausEDROPDataAdapter.doStart(SpamhausEDROPDataAdapter.java:68) ~[?:?]
	at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
	at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
	at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2018-09-27T09:36:50.224+02:00 INFO  [LookupTableService] Data Adapter abuse-ch-ransomware-ip/5a53280a3656ae5184db21fc [@27ce9c71] STARTING
2018-09-27T09:36:50.225+02:00 ERROR [LookupDataAdapter] Couldn't start data adapter <abuse-ch-ransomware-ip/5a53280a3656ae5184db21fc/@27ce9c71>
org.graylog.plugins.threatintel.tools.AdapterDisabledException: Abuse.ch service is disabled, not starting adapter. To enable it please go to System / Configurations.
	at org.graylog.plugins.threatintel.adapters.abusech.AbuseChRansomAdapter.doStart(AbuseChRansomAdapter.java:80) ~[?:?]
	at org.graylog2.plugin.lookup.LookupDataAdapter.startUp(LookupDataAdapter.java:59) [graylog.jar:?]
	at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
	at com.google.common.util.concurrent.Callables$4.run(Callables.java:122) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2018-09-27T09:36:50.228+02:00 INFO  [LookupTableService] Data Adapter tor-exit-node/5a53280a3656ae5184db21fe [@21c4e6fc] RUNNING
2018-09-27T09:36:50.230+02:00 INFO  [LookupTableService] Data Adapter spamhaus-drop/5a53280a3656ae5184db2200 [@3fccbf84] RUNNING
2018-09-27T09:36:50.237+02:00 INFO  [LookupTableService] Data Adapter otx-api-domain/5a53280a3656ae5184db2201 [@27b70cec] RUNNING
2018-09-27T09:36:50.237+02:00 INFO  [LookupTableService] Data Adapter otx-api-ip/5a53280a3656ae5184db21ff [@530ac3] RUNNING
2018-09-27T09:36:50.235+02:00 INFO  [LookupTableService] Data Adapter abuse-ch-ransomware-ip/5a53280a3656ae5184db21fc [@27ce9c71] RUNNING
2018-09-27T09:36:50.254+02:00 INFO  [LookupTableService] Cache threat-intel-uncached-adapters/5a53280a3656ae5184db21f7 [@45179183] STARTING
2018-09-27T09:36:50.254+02:00 INFO  [LookupTableService] Cache whois-cache/5a53280a3656ae5184db21f9 [@3e79bf1d] STARTING
2018-09-27T09:36:50.256+02:00 INFO  [LookupTableService] Cache threat-intel-uncached-adapters/5a53280a3656ae5184db21f7 [@45179183] RUNNING
2018-09-27T09:36:50.256+02:00 INFO  [LookupTableService] Cache otx-api-ip-cache/5a53280a3656ae5184db21f8 [@70012117] STARTING
2018-09-27T09:36:50.261+02:00 INFO  [LookupTableService] Cache otx-api-ip-cache/5a53280a3656ae5184db21f8 [@70012117] RUNNING
2018-09-27T09:36:50.261+02:00 INFO  [LookupTableService] Cache whois-cache/5a53280a3656ae5184db21f9 [@3e79bf1d] RUNNING
2018-09-27T09:36:50.261+02:00 INFO  [LookupTableService] Cache spamhaus-e-drop-cache/5a53280a3656ae5184db21fa [@1b71e1ba] STARTING
2018-09-27T09:36:50.258+02:00 INFO  [LookupTableService] Cache otx-api-domain-cache/5a53280a3656ae5184db21f6 [@58a8b76f] STARTING
2018-09-27T09:36:50.262+02:00 INFO  [LookupTableService] Cache spamhaus-e-drop-cache/5a53280a3656ae5184db21fa [@1b71e1ba] RUNNING
2018-09-27T09:36:50.262+02:00 INFO  [LookupTableService] Cache otx-api-domain-cache/5a53280a3656ae5184db21f6 [@58a8b76f] RUNNING
2018-09-27T09:36:50.271+02:00 INFO  [LookupTableService] Starting lookup table tor-exit-node-list/5a53280a3656ae5184db2204 [@492aef31] using cache threat-intel-uncached-adapters/5a53280a3656ae5184db21f7 [@45179183], data adapter tor-exit-node/5a53280a3656ae5184db21fe [@21c4e6fc]
2018-09-27T09:36:50.272+02:00 INFO  [LookupTableService] Starting lookup table whois/5a53280a3656ae5184db2205 [@6457e802] using cache whois-cache/5a53280a3656ae5184db21f9 [@3e79bf1d], data adapter whois/5a53280a3656ae5184db2202 [@ac8a9]
2018-09-27T09:36:50.272+02:00 INFO  [LookupTableService] Starting lookup table otx-api-domain/5a53280a3656ae5184db2206 [@4fadcadb] using cache otx-api-domain-cache/5a53280a3656ae5184db21f6 [@58a8b76f], data adapter otx-api-domain/5a53280a3656ae5184db2201 [@27b70cec]
2018-09-27T09:36:50.272+02:00 INFO  [LookupTableService] Starting lookup table abuse-ch-ransomware-domains/5a53280a3656ae5184db2207 [@11fb8846] using cache threat-intel-uncached-adapters/5a53280a3656ae5184db21f7 [@45179183], data adapter abuse-ch-ransomware-domains/5a53280a3656ae5184db21fd [@1464d747]
2018-09-27T09:36:50.272+02:00 INFO  [LookupTableService] Starting lookup table spamhaus-drop/5a53280a3656ae5184db2208 [@fb6ae2f] using cache spamhaus-e-drop-cache/5a53280a3656ae5184db21fa [@1b71e1ba], data adapter spamhaus-drop/5a53280a3656ae5184db2200 [@3fccbf84]
2018-09-27T09:36:50.273+02:00 INFO  [LookupTableService] Starting lookup table otx-api-ip/5a53280a3656ae5184db2209 [@547a5023] using cache otx-api-ip-cache/5a53280a3656ae5184db21f8 [@70012117], data adapter otx-api-ip/5a53280a3656ae5184db21ff [@530ac3]
2018-09-27T09:36:50.273+02:00 INFO  [LookupTableService] Starting lookup table abuse-ch-ransomware-ip/5a53280a3656ae5184db220a [@7520b365] using cache threat-intel-uncached-adapters/5a53280a3656ae5184db21f7 [@45179183], data adapter abuse-ch-ransomware-ip/5a53280a3656ae5184db21fc [@27ce9c71]
2018-09-27T09:36:50.625+02:00 INFO  [JerseyService] Enabling CORS for HTTP endpoint
2018-09-27T09:37:02.759+02:00 INFO  [NetworkListener] Started listener bound to [10.4.1.214:12900]
2018-09-27T09:37:02.761+02:00 INFO  [HttpServer] [HttpServer] Started.
2018-09-27T09:37:02.762+02:00 INFO  [JerseyService] Started REST API at <http://10.4.1.214:12900/>
2018-09-27T09:37:06.547+02:00 INFO  [NetworkListener] Started listener bound to [127.0.0.1:9000]
2018-09-27T09:37:06.547+02:00 INFO  [HttpServer] [HttpServer-1] Started.
2018-09-27T09:37:06.548+02:00 INFO  [JerseyService] Started Web Interface at <http://127.0.0.1:9000/>
2018-09-27T09:37:06.548+02:00 INFO  [ServiceManagerListener] Services are healthy
2018-09-27T09:37:06.549+02:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2018-09-27T09:37:06.551+02:00 INFO  [ServerBootstrap] Services started, startup times in ms: {InputSetupService [RUNNING]=8, OutputSetupService [RUNNING]=9, ConfigurationEtagService [RUNNING]=9, BufferSynchronizerService [RUNNING]=13, JournalReader [RUNNING]=22, KafkaJournal [RUNNING]=25, PeriodicalsService [RUNNING]=55, StreamCacheService [RUNNING]=117, LookupTableService [RUNNING]=156, JerseyService [RUNNING]=16440}
2018-09-27T09:37:06.561+02:00 INFO  [ServerBootstrap] Graylog server up and running.
2018-09-27T09:37:06.610+02:00 INFO  [InputStateListener] Input [Syslog UDP/57ceacb7e40ee75636e93f05] is now STARTING
2018-09-27T09:37:06.612+02:00 INFO  [InputStateListener] Input [GELF UDP/58e4b752bab9aa05c9e4dd10] is now STARTING
2018-09-27T09:37:06.615+02:00 INFO  [InputStateListener] Input [Syslog UDP/58e4ba31bab9aa05c9e4e049] is now STARTING
2018-09-27T09:37:06.616+02:00 INFO  [InputStateListener] Input [Syslog UDP/58f9d731bab9aa05c9fb10ff] is now STARTING
2018-09-27T09:37:06.619+02:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/58fef44bbab9aa05c9007083] is now STARTING
2018-09-27T09:37:06.621+02:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/58fef572bab9aa05c90071c3] is now STARTING
2018-09-27T09:37:06.622+02:00 INFO  [InputStateListener] Input [GELF TCP/58ff3a0ebab9aa05c900ba34] is now STARTING
2018-09-27T09:37:06.623+02:00 INFO  [InputStateListener] Input [Syslog UDP/59243bee3656ae03347a15a9] is now STARTING
2018-09-27T09:37:06.625+02:00 INFO  [InputStateListener] Input [Beats/59a3ddfb3656ae188faee278] is now STARTING
2018-09-27T09:37:06.626+02:00 INFO  [InputStateListener] Input [Beats/59a026163656ae188faaf923] is now STARTING
2018-09-27T09:37:06.627+02:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/59c9f6d328bd740abfef05fa] is now STARTING
2018-09-27T09:37:06.629+02:00 INFO  [InputStateListener] Input [Beats/59d2252d3656ae6c891f2b62] is now STARTING
2018-09-27T09:37:06.631+02:00 INFO  [InputStateListener] Input [Beats/5a180bfc3656ae3bd815d02c] is now STARTING
2018-09-27T09:37:06.632+02:00 INFO  [InputStateListener] Input [GELF UDP/5a33e52328f8af3e408dd23e] is now STARTING
2018-09-27T09:37:06.633+02:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5a5de8a628f8af1b134ac131] is now STARTING
2018-09-27T09:37:06.637+02:00 INFO  [InputStateListener] Input [Beats/5a33a25628f8af3e408d8f02] is now STARTING
2018-09-27T09:37:06.638+02:00 INFO  [InputStateListener] Input [Beats/5a158fdd3656ae3bd8131e12] is now STARTING
2018-09-27T09:37:06.639+02:00 INFO  [InputStateListener] Input [Syslog UDP/5abceaf228bd740466e8e40d] is now STARTING
2018-09-27T09:37:06.641+02:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5adefaac1fec54166f8755b6] is now STARTING
2018-09-27T09:37:07.505+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=RAW udp 514 vers 8514, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=null} should be 262144 but is 212992.
2018-09-27T09:37:07.505+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=gelf udp global, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} should be 262144 but is 212992.
2018-09-27T09:37:07.508+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=raw plaintext haproxy http global, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=null} should be 1048576 but is 212992.
2018-09-27T09:37:07.508+02:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5adefaac1fec54166f8755b6] is now RUNNING
2018-09-27T09:37:07.510+02:00 INFO  [InputStateListener] Input [GELF UDP/58e4b752bab9aa05c9e4dd10] is now RUNNING
2018-09-27T09:37:07.511+02:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/58fef44bbab9aa05c9007083] is now RUNNING
2018-09-27T09:37:07.521+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=syslog sam et janice global, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} should be 262144 but is 212992.
2018-09-27T09:37:07.522+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=cisco udp 13514 global, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} should be 262144 but is 212992.
2018-09-27T09:37:07.523+02:00 INFO  [InputStateListener] Input [Syslog UDP/58f9d731bab9aa05c9fb10ff] is now RUNNING
2018-09-27T09:37:07.528+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input BeatsInput{title=Beats 5044 global, type=org.graylog.plugins.beats.BeatsInput, nodeId=null} should be 1048576 but is 212992.
2018-09-27T09:37:07.533+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input BeatsInput{title=beats 5099 cluster recherche global, type=org.graylog.plugins.beats.BeatsInput, nodeId=null} should be 1048576 but is 212992.
2018-09-27T09:37:07.535+02:00 INFO  [InputStateListener] Input [Syslog UDP/59243bee3656ae03347a15a9] is now RUNNING
2018-09-27T09:37:07.540+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input BeatsInput{title=beats 5025 zimbra global, type=org.graylog.plugins.beats.BeatsInput, nodeId=null} should be 1048576 but is 212992.
2018-09-27T09:37:07.541+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=fortinet udp RAW 11514 global, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=null} should be 262144 but is 212992.
2018-09-27T09:37:07.541+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=fortinet RAW udp 12514 global, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=null} should be 262144 but is 212992.
2018-09-27T09:37:07.545+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=gelf udp 12225 zimbra nginx, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=null} should be 262144 but is 212992.
2018-09-27T09:37:07.542+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=syslog udp 10514 global, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} should be 262144 but is 212992.
2018-09-27T09:37:07.548+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input BeatsInput{title=beats 5026 zimbra nginx global, type=org.graylog.plugins.beats.BeatsInput, nodeId=null} should be 1048576 but is 212992.
2018-09-27T09:37:07.554+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input BeatsInput{title=Beats 5046 global (Tomcat), type=org.graylog.plugins.beats.BeatsInput, nodeId=null} should be 1048576 but is 212992.
2018-09-27T09:37:07.554+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=raw plaintext haproxy tcp global, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=null} should be 262144 but is 212992.
2018-09-27T09:37:07.554+02:00 INFO  [InputStateListener] Input [Beats/59a3ddfb3656ae188faee278] is now RUNNING
2018-09-27T09:37:07.549+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=switchs syslog udp 15514, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} should be 262144 but is 212992.
2018-09-27T09:37:07.560+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input GELFTCPInput{title=gelf tcp12203 pour apache global, type=org.graylog2.inputs.gelf.tcp.GELFTCPInput, nodeId=null} should be 1048576 but is 212992.
2018-09-27T09:37:07.561+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=Extreme-wifi 14514 syslog udp, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=eb99cbd2-552c-4272-aae7-b305accbb2dd} should be 262144 but is 212992.
2018-09-27T09:37:07.566+02:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input BeatsInput{title=Beats 5045 global, type=org.graylog.plugins.beats.BeatsInput, nodeId=null} should be 1048576 but is 212992.
2018-09-27T09:37:07.568+02:00 INFO  [InputStateListener] Input [Beats/5a180bfc3656ae3bd815d02c] is now RUNNING
2018-09-27T09:37:07.570+02:00 INFO  [InputStateListener] Input [Beats/5a158fdd3656ae3bd8131e12] is now RUNNING
2018-09-27T09:37:07.571+02:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/5a5de8a628f8af1b134ac131] is now RUNNING
2018-09-27T09:37:07.572+02:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/59c9f6d328bd740abfef05fa] is now RUNNING
2018-09-27T09:37:07.573+02:00 INFO  [InputStateListener] Input [GELF UDP/5a33e52328f8af3e408dd23e] is now RUNNING
2018-09-27T09:37:07.575+02:00 INFO  [InputStateListener] Input [Syslog UDP/58e4ba31bab9aa05c9e4e049] is now RUNNING
2018-09-27T09:37:07.576+02:00 INFO  [InputStateListener] Input [Beats/5a33a25628f8af3e408d8f02] is now RUNNING
2018-09-27T09:37:07.577+02:00 INFO  [InputStateListener] Input [Beats/59d2252d3656ae6c891f2b62] is now RUNNING
2018-09-27T09:37:07.578+02:00 INFO  [InputStateListener] Input [Raw/Plaintext UDP/58fef572bab9aa05c90071c3] is now RUNNING
2018-09-27T09:37:07.580+02:00 INFO  [InputStateListener] Input [Syslog UDP/5abceaf228bd740466e8e40d] is now RUNNING
2018-09-27T09:37:07.581+02:00 INFO  [InputStateListener] Input [GELF TCP/58ff3a0ebab9aa05c900ba34] is now RUNNING
2018-09-27T09:37:07.582+02:00 INFO  [InputStateListener] Input [Syslog UDP/57ceacb7e40ee75636e93f05] is now RUNNING
2018-09-27T09:37:07.584+02:00 INFO  [InputStateListener] Input [Beats/59a026163656ae188faaf923] is now RUNNING

sdu · September 27, 2018, 12:46pm

however, i got recurrent errors like that :

2018-08-31T15:00:54.621+02:00 ERROR [AlertConditionFactory] Could not load alert condition <95f950d4-2b2b-47a1-95b5-a23f8bec08f6>, invalid configuration detected.
2018-08-31T15:00:54.621+02:00 ERROR [StreamServiceImpl] Skipping alert condition.
org.graylog2.plugin.configuration.ConfigurationException: Mandatory configuration field backlog is missing or has the wrong data type
        at org.graylog2.plugin.configuration.ConfigurationRequest.check(ConfigurationRequest.java:111) ~[graylog.jar:?]
        at org.graylog2.alerts.AlertConditionFactory.createAlertCondition(AlertConditionFactory.java:63) ~[graylog.jar:?]
        at org.graylog2.alerts.AlertServiceImpl.fromPersisted(AlertServiceImpl.java:170) ~[graylog.jar:?]
        at org.graylog2.streams.StreamServiceImpl.getAlertConditions(StreamServiceImpl.java:294) [graylog.jar:?]
        at org.graylog2.periodical.AlertScannerThread.doRun(AlertScannerThread.java:55) [graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_181]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

which I think is linked to another problem.

jan · September 27, 2018, 1:37pm

does the Graylog node do any processing while hitting 100% or is this while doing nothing?

sdu · September 27, 2018, 2:00pm

it is while doing nothing. The load balancer doesn’t send any log - I paused message processing on that node to be sure.

top :

30706 graylog   20   0 16,493g 1,706g  25016 S  1102 10,9   4190:02 /usr/bin/java -Xms2g -Xmx2g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -jar -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/sh+

node state from de gui :

 4b3eab03 / zabbix-ng.insa-rennes.fr In 0 / Out 0 msg/s.
The journal contains 0 unprocessed messages in 1 segment. 0 messages appended, 0 messages read in the last second.
 4b3eab03 / zabbix-ng.insa-rennes.fr In 0 / Out 0 msg/s.
The journal contains 0 unprocessed messages in 1 segment. 0 messages appended, 0 messages read in the last second.

Current lifecycle state:
    Paused
Message processing:
    Disabled
Load balancer indication:
    ALIVE

sdu · September 27, 2018, 2:23pm

server.conf on that node :

is_master = false
node_id_file = /etc/graylog/server/node-id
password_secret = ***
root_password_sha2 = ***
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://10.4.1.214:12900/
rest_transport_uri = http://10.4.1.214:12900/
elasticsearch_hosts = http://10.4.1.57:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = true
allow_highlighting = true
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = yielding
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = yielding
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://sonde-bdd/graylog2
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks
content_packs_auto_load = grok-patterns.json
proxied_requests_thread_pool_size = 32
transport_email_enabled = true
transport_email_hostname = smtp.insa-rennes.fr
transport_email_port = 25
transport_email_use_auth = false
transport_email_use_tls = false
transport_email_use_ssl = false
transport_email_auth_username = you@example.com
transport_email_auth_password = secret
transport_email_subject_prefix = [graylog2]
transport_email_from_email = cri-systeme@insa-rennes.fr
transport_email_from_name = Graylog2
transport_email_web_interface_url = http://sonde.insa-rennes.fr:9000
http_write_timeout = 20s
message_cache_spool_dir = /var/lib/graylog2-server/message-cache-spool
ldap_connection_timeout = 2000

sdu · October 3, 2018, 7:46am

update :
a tcpdump give alot of mongo 27017 traffic :

09:32:55.359760 IP 10.4.1.215.55466 > 10.4.1.214.12900: Flags [P.], seq 1504:1973, ack 2131, win 1444, options [nop,nop,TS val 2130363701 ecr 1646958687], length 469
09:32:55.360077 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 28806:28962, ack 142317, win 4705, options [nop,nop,TS val 2915888533 ecr 3761703252], length 156
09:32:55.360302 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 142317:142482, ack 28962, win 1452, options [nop,nop,TS val 3761703286 ecr 2915888533], length 165
09:32:55.360327 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [.], ack 142482, win 4705, options [nop,nop,TS val 2915888533 ecr 3761703286], length 0
09:32:55.360628 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 28962:29119, ack 142482, win 4705, options [nop,nop,TS val 2915888533 ecr 3761703286], length 157
09:32:55.360941 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 142482:143383, ack 29119, win 1452, options [nop,nop,TS val 3761703286 ecr 2915888533], length 901
09:32:55.361164 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 29119:29276, ack 143383, win 4705, options [nop,nop,TS val 2915888534 ecr 3761703286], length 157
09:32:55.361485 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 143383:144284, ack 29276, win 1452, options [nop,nop,TS val 3761703287 ecr 2915888534], length 901
09:32:55.361699 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 29276:29433, ack 144284, win 4705, options [nop,nop,TS val 2915888534 ecr 3761703287], length 157
09:32:55.362019 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 144284:145185, ack 29433, win 1452, options [nop,nop,TS val 3761703287 ecr 2915888534], length 901
09:32:55.362231 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 29433:29590, ack 145185, win 4705, options [nop,nop,TS val 2915888535 ecr 3761703287], length 157
09:32:55.362547 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 145185:146086, ack 29590, win 1452, options [nop,nop,TS val 3761703287 ecr 2915888535], length 901
09:32:55.362755 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 29590:29747, ack 146086, win 4705, options [nop,nop,TS val 2915888535 ecr 3761703287], length 157
09:32:55.363075 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 146086:146987, ack 29747, win 1452, options [nop,nop,TS val 3761703287 ecr 2915888535], length 901
09:32:55.363283 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 29747:29904, ack 146987, win 4705, options [nop,nop,TS val 2915888536 ecr 3761703287], length 157
09:32:55.363604 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 146987:147888, ack 29904, win 1452, options [nop,nop,TS val 3761703287 ecr 2915888536], length 901
09:32:55.363819 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 29904:30061, ack 147888, win 4705, options [nop,nop,TS val 2915888536 ecr 3761703287], length 157
09:32:55.364048 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 147888:148789, ack 30061, win 1452, options [nop,nop,TS val 3761703287 ecr 2915888536], length 901
09:32:55.364258 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 30061:30218, ack 148789, win 4705, options [nop,nop,TS val 2915888537 ecr 3761703287], length 157
09:32:55.364518 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 148789:149690, ack 30218, win 1452, options [nop,nop,TS val 3761703287 ecr 2915888537], length 901
09:32:55.364725 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 30218:30375, ack 149690, win 4705, options [nop,nop,TS val 2915888537 ecr 3761703287], length 157
09:32:55.364971 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 149690:150591, ack 30375, win 1452, options [nop,nop,TS val 3761703287 ecr 2915888537], length 901
09:32:55.365180 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 30375:30532, ack 150591, win 4705, options [nop,nop,TS val 2915888538 ecr 3761703287], length 157
09:32:55.365429 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 150591:151492, ack 30532, win 1452, options [nop,nop,TS val 3761703288 ecr 2915888538], length 901
09:32:55.365636 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 30532:30628, ack 151492, win 4705, options [nop,nop,TS val 2915888538 ecr 3761703288], length 96
09:32:55.365872 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 151492:152094, ack 30628, win 1452, options [nop,nop,TS val 3761703288 ecr 2915888538], length 602
09:32:55.366093 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 30628:30785, ack 152094, win 4705, options [nop,nop,TS val 2915888539 ecr 3761703288], length 157
09:32:55.366357 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 152094:152995, ack 30785, win 1452, options [nop,nop,TS val 3761703288 ecr 2915888539], length 901
09:32:55.366569 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 30785:30942, ack 152995, win 4705, options [nop,nop,TS val 2915888539 ecr 3761703288], length 157
09:32:55.366814 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 152995:153896, ack 30942, win 1452, options [nop,nop,TS val 3761703288 ecr 2915888539], length 901
09:32:55.367091 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 30942:31099, ack 153896, win 4705, options [nop,nop,TS val 2915888540 ecr 3761703288], length 157
09:32:55.367339 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 153896:154797, ack 31099, win 1452, options [nop,nop,TS val 3761703288 ecr 2915888540], length 901
09:32:55.367549 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 31099:31256, ack 154797, win 4705, options [nop,nop,TS val 2915888540 ecr 3761703288], length 157
09:32:55.367792 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 154797:155698, ack 31256, win 1452, options [nop,nop,TS val 3761703288 ecr 2915888540], length 901
09:32:55.368135 IP 10.4.1.214.12900 > 10.4.1.215.55466: Flags [P.], seq 2131:2595, ack 1973, win 1452, options [nop,nop,TS val 1646960022 ecr 2130363701], length 464

and then

09:32:56.110683 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 31256:31407, ack 155698, win 4705, options [nop,nop,TS val 2915889283 ecr 3761703288], length 151
09:32:56.111048 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 155698:156022, ack 31407, win 1452, options [nop,nop,TS val 3761703474 ecr 2915889283], length 324
09:32:56.111065 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [.], ack 156022, win 4705, options [nop,nop,TS val 2915889284 ecr 3761703474], length 0
09:32:56.111208 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 31407:31744, ack 156022, win 4705, options [nop,nop,TS val 2915889284 ecr 3761703474], length 337
09:32:56.111477 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 156022:156097, ack 31744, win 1452, options [nop,nop,TS val 3761703474 ecr 2915889284], length 75
09:32:56.111559 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 31744:31889, ack 156097, win 4705, options [nop,nop,TS val 2915889284 ecr 3761703474], length 145
09:32:56.111795 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 156097:156157, ack 31889, win 1452, options [nop,nop,TS val 3761703474 ecr 2915889284], length 60
09:32:56.111901 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 31889:32024, ack 156157, win 4705, options [nop,nop,TS val 2915889285 ecr 3761703474], length 135
09:32:56.112121 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 156157:156478, ack 32024, win 1452, options [nop,nop,TS val 3761703474 ecr 2915889285], length 321
09:32:56.112208 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 32024:32168, ack 156478, win 4705, options [nop,nop,TS val 2915889285 ecr 3761703474], length 144
09:32:56.112471 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 156478:156538, ack 32168, win 1452, options [nop,nop,TS val 3761703474 ecr 2915889285], length 60
09:32:56.114507 IP 10.4.1.214.40840 > 10.4.1.57.27017: Flags [P.], seq 32168:32350, ack 156538, win 4705, options [nop,nop,TS val 2915889287 ecr 3761703474], length 182
09:32:56.114810 IP 10.4.1.57.27017 > 10.4.1.214.40840: Flags [P.], seq 156538:156665, ack 32350, win 1452, options [nop,nop,TS val 3761703475 ecr 2915889287], length 127

and repeat

so, maybe it’s linked to mongodb? but why?

anyway thanx, jan for taking your time reading this post :
Keep up, the good work at graylog’s
cheers

jan · October 3, 2018, 10:30am

MongoDB holds the configuration and also some of the error logs - so if an error is given, you get lots of MongoDB traffic …

Do you have lots of alerts configured? If not - just drop them all and look if the error is persistent. If yes then we would need to drop the alert configuration from MongoDB after you had shutdown Graylog.

sdu · October 3, 2018, 2:00pm

thanx, will try this and post an update though.

Thanx again.

system · October 17, 2018, 2:09pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
High CPU on GrayLog Nodes Graylog Central (peer support)	7	7998	December 19, 2018
CPU max on single Node Cluster Graylog Central (peer support)	3	417	September 24, 2023
High CPU load on a three node cluster Graylog Central (peer support)	7	1562	March 17, 2022
Graylog.cluster_config high latency in Production stage Graylog Central (peer support)	2	228	August 15, 2022
Master/Leader node disappearing with high CPU on Java Graylog Central (peer support)	10	804	September 1, 2022

Graylog cluster : high cpu on a node even if there's no active input nor processing

Related Topics