Graylog not processing to Elasticseach anymore

chas0rde · September 13, 2022, 2:46pm

Out graylog instance stopped processing after ubuntu release upgrade from 20.04LTS to 22.04LTS

From my point of view versions are ok.
We have two systems (1.Graylog+MongoDB and 2. Elasticsearch)
Graylog is Graylog 4.3.6+36120cf (behind apache2 as https proxy)
Elasticsearch is 7.10.2

Processbuffer is full and Journal keeps growing:

/etc/graylog/server/server.conf (comments and commented-out parameters removed)

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = <somepassword>
root_password_sha2 = <somepassword>
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = 0.0.0.0:9000
http_external_uri = https://graylog.monitoring.somedomain.local/
elasticsearch_hosts = http://elastic:9200
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = true
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 8
outputbuffer_processors = 5
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
message_journal_max_age = 48h
message_journal_max_size = 20gb
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32
elasticsearch_mute_deprecation_warnings=false

/etc/elasticsearch/elasticsearch.yml

cluster.name: graylog
path.data: /graylog-data/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
discovery.type: single-node
xpack.security.enabled: false
action.auto_create_index: .watches,.triggered_watches,.watcher-history-*

chas0rde · September 13, 2022, 2:47pm

Some logs of both services starting. Elasticsearch shows health to be green (see log below)
/var/log/graylog-server/server.log



2022-09-13T16:44:57.497+02:00 INFO  [ImmutableFeatureFlagsCollector] Following feature flags are used: {}
2022-09-13T16:44:58.029+02:00 INFO  [CmdLineTool] Loaded plugin: AWS plugins 4.3.6 [org.graylog.aws.AWSPlugin]
2022-09-13T16:44:58.029+02:00 INFO  [CmdLineTool] Loaded plugin: Enterprise Integrations 4.3.6 [org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin]
2022-09-13T16:44:58.030+02:00 INFO  [CmdLineTool] Loaded plugin: Integrations 4.3.6 [org.graylog.integrations.IntegrationsPlugin]
2022-09-13T16:44:58.031+02:00 INFO  [CmdLineTool] Loaded plugin: Collector 4.3.6 [org.graylog.plugins.collector.CollectorPlugin]
2022-09-13T16:44:58.031+02:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise 4.3.6 [org.graylog.plugins.enterprise.EnterprisePlugin]
2022-09-13T16:44:58.032+02:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES6 Support) 4.3.6 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es6.EnterpriseES6Plugin]
2022-09-13T16:44:58.032+02:00 INFO  [CmdLineTool] Loaded plugin: Graylog Enterprise (ES7 Support) 4.3.6 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es7.EnterpriseES7Plugin]
2022-09-13T16:44:58.033+02:00 INFO  [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.3.6 [org.graylog.plugins.threatintel.ThreatIntelPlugin]
2022-09-13T16:44:58.033+02:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.3.6+36120cf [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin]
2022-09-13T16:44:58.033+02:00 INFO  [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.3.6+36120cf [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin]
2022-09-13T16:44:58.055+02:00 INFO  [CmdLineTool] Running with JVM arguments: -Xms8g -Xmx8g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2022-09-13T16:44:58.312+02:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2022-09-13T16:44:58.338+02:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-09-13T16:44:58.350+02:00 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:109}] to localhost:27017
2022-09-13T16:44:58.353+02:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 16]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2006759}
2022-09-13T16:44:58.362+02:00 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:110}] to localhost:27017
2022-09-13T16:44:58.378+02:00 INFO  [connection] Closed connection [connectionId{localValue:2, serverValue:110}] to localhost:27017 because the pool has been closed.
2022-09-13T16:44:58.380+02:00 INFO  [MongoDBPreflightCheck] Connected to MongoDB version 4.4.16
2022-09-13T16:44:58.547+02:00 INFO  [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <Elasticsearch:7.10.2>
2022-09-13T16:44:58.642+02:00 INFO  [Version] HV000001: Hibernate Validator null
2022-09-13T16:45:00.734+02:00 INFO  [InputBufferImpl] Message journal is enabled.
2022-09-13T16:45:00.746+02:00 INFO  [NodeId] Node ID: 327f2801-d3ee-4215-bfe3-36016b482b56
2022-09-13T16:45:00.868+02:00 INFO  [LogManager] Loading logs.
2022-09-13T16:45:01.063+02:00 INFO  [LogManager] Logs loading complete.
2022-09-13T16:45:01.065+02:00 INFO  [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2022-09-13T16:45:01.069+02:00 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2022-09-13T16:45:01.071+02:00 INFO  [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out
2022-09-13T16:45:01.072+02:00 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:111}] to localhost:27017
2022-09-13T16:45:01.073+02:00 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 16]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=529289}
2022-09-13T16:45:01.080+02:00 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:112}] to localhost:27017
2022-09-13T16:45:01.188+02:00 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2022-09-13T16:45:01.308+02:00 INFO  [ElasticsearchVersionProvider] Elasticsearch cluster is running Elasticsearch:7.10.2
2022-09-13T16:45:01.895+02:00 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-09-13T16:45:02.087+02:00 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2022-09-13T16:45:02.149+02:00 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:113}] to localhost:27017
2022-09-13T16:45:02.676+02:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-09-13T16:45:02.677+02:00 INFO  [ServerBootstrap] Graylog server 4.3.6+36120cf starting up
2022-09-13T16:45:02.678+02:00 INFO  [ServerBootstrap] JRE: Ubuntu 11.0.16 on Linux 5.15.0-47-generic
2022-09-13T16:45:02.678+02:00 INFO  [ServerBootstrap] Deployment: deb
2022-09-13T16:45:02.678+02:00 INFO  [ServerBootstrap] OS: Ubuntu 22.04.1 LTS (jammy)
2022-09-13T16:45:02.678+02:00 INFO  [ServerBootstrap] Arch: amd64
2022-09-13T16:45:02.769+02:00 INFO  [ServerBootstrap] Running 62 migrations...
2022-09-13T16:45:03.242+02:00 INFO  [PeriodicalsService] Starting 37 periodicals ...
2022-09-13T16:45:03.245+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2022-09-13T16:45:03.246+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2022-09-13T16:45:03.247+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node.
2022-09-13T16:45:03.247+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2022-09-13T16:45:03.247+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2022-09-13T16:45:03.248+02:00 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node.
2022-09-13T16:45:03.248+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2022-09-13T16:45:03.249+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2022-09-13T16:45:03.249+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexBlockCheck] periodical in [0s], polling every [30s].
2022-09-13T16:45:03.250+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2022-09-13T16:45:03.250+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2022-09-13T16:45:03.251+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2022-09-13T16:45:03.251+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2022-09-13T16:45:03.251+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2022-09-13T16:45:03.252+02:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2022-09-13T16:45:03.253+02:00 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2022-09-13T16:45:03.303+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2022-09-13T16:45:03.303+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2022-09-13T16:45:03.304+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s].
2022-09-13T16:45:03.306+02:00 INFO  [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [1s].
2022-09-13T16:45:03.305+02:00 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2022-09-13T16:45:03.308+02:00 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:114}] to localhost:27017
2022-09-13T16:45:03.308+02:00 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:115}] to localhost:27017
2022-09-13T16:45:03.307+02:00 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:116}] to localhost:27017
2022-09-13T16:45:03.308+02:00 INFO  [connection] Opened connection [connectionId{localValue:9, serverValue:117}] to localhost:27017
2022-09-13T16:45:03.309+02:00 INFO  [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s].
2022-09-13T16:45:03.310+02:00 INFO  [connection] Opened connection [connectionId{localValue:11, serverValue:118}] to localhost:27017
2022-09-13T16:45:03.310+02:00 INFO  [connection] Opened connection [connectionId{localValue:10, serverValue:119}] to localhost:27017
2022-09-13T16:45:03.310+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s].
2022-09-13T16:45:03.313+02:00 INFO  [Periodicals] Starting [org.graylog2.periodical.UserSessionTerminationPeriodical] periodical, running forever.
2022-09-13T16:45:03.315+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s].
2022-09-13T16:45:03.316+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s].
2022-09-13T16:45:03.319+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s].
2022-09-13T16:45:03.319+02:00 INFO  [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s].
2022-09-13T16:45:03.320+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2022-09-13T16:45:03.321+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.forwarder.ForwarderStatePeriodical] periodical in [0s], polling every [15s].
2022-09-13T16:45:03.324+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.license.LicenseManagerPeriodical] periodical in [0s], polling every [300s].
2022-09-13T16:45:03.328+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.license.LicenseReportPeriodical] periodical in [300s], polling every [3600s].
2022-09-13T16:45:03.328+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.license.StagedLicenseInstallerPeriodical] periodical, running forever.
2022-09-13T16:45:03.328+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.auditlog.mongodb.MongoAuditLogPeriodical] periodical in [0s], polling every [3600s].
2022-09-13T16:45:03.329+02:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license.
2022-09-13T16:45:03.329+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.report.scheduler.ReportPeriodical] periodical in [120s], polling every [60s].
2022-09-13T16:45:03.329+02:00 INFO  [Periodicals] Starting [org.graylog.plugins.files.CleanupPeriodical] periodical in [0s], polling every [86400s].
2022-09-13T16:45:03.330+02:00 INFO  [PeriodicalsService] Not starting [org.graylog.plugins.securityapp.anomaly.retrieval.AnomalyRetrievalPeriodical] periodical. Not configured to run on this node.
2022-09-13T16:45:03.330+02:00 INFO  [PeriodicalsService] Not starting [org.graylog.plugins.securityapp.anomaly.DetectorStatusSyncPeriodical] periodical. Not configured to run on this node.
2022-09-13T16:45:03.338+02:00 WARN  [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-09-13T16:45:03.341+02:00 ERROR [GeoIpDbFileChangeMonitorService] Error validating GeoIP Database files. Invalid 'MAXMIND' City Geo IP database file '/etc/graylog/server/GeoLite2-City.mmdb'.  Make sure the file exists and is valid for 'MAXMIND'
org.graylog2.plugin.validate.ConfigValidationException: Invalid 'MAXMIND' City Geo IP database file '/etc/graylog/server/GeoLite2-City.mmdb'.  Make sure the file exists and is valid for 'MAXMIND'
        at org.graylog2.rest.resources.system.GeoIpResolverConfigValidator.validateConfig(GeoIpResolverConfigValidator.java:93) ~[graylog.jar:?]
        at org.graylog2.rest.resources.system.GeoIpResolverConfigValidator.validate(GeoIpResolverConfigValidator.java:65) ~[graylog.jar:?]
        at org.graylog.plugins.map.geoip.GeoIpDbFileChangeMonitorService.updateConfiguration(GeoIpDbFileChangeMonitorService.java:147) [graylog.jar:?]
        at org.graylog.plugins.map.geoip.GeoIpDbFileChangeMonitorService.startUp(GeoIpDbFileChangeMonitorService.java:97) [graylog.jar:?]
        at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?]
        at com.google.common.util.concurrent.Callables$4.run(Callables.java:121) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
2022-09-13T16:45:03.364+02:00 INFO  [LookupTableService] Data Adapter watchlist-mongo/63203987d002820008fe8c8f [@16ea03b4] STARTING
2022-09-13T16:45:03.364+02:00 INFO  [LookupTableService] Data Adapter dns-lookup-internal/60f9dc141a6c676f0bc97372 [@79e118bb] STARTING
2022-09-13T16:45:03.364+02:00 INFO  [LookupTableService] Data Adapter otx-api-ip/60c35d1ed7ae89169c78f333 [@cfc2c40] STARTING
2022-09-13T16:45:03.364+02:00 INFO  [LookupTableService] Data Adapter whois/60c35d29d7ae89169c78f34c [@53f14fd] STARTING
2022-09-13T16:45:03.364+02:00 WARN  [OTXDataAdapter] OTX API key is missing. Make sure to add the key to allow higher request limits.
2022-09-13T16:45:03.366+02:00 INFO  [LookupTableService] Data Adapter otx-api-domain/60c35d1ed7ae89169c78f335 [@45f7eb77] STARTING
2022-09-13T16:45:03.366+02:00 INFO  [LookupTableService] Data Adapter watchlist-mongo/63203987d002820008fe8c8f [@16ea03b4] RUNNING
2022-09-13T16:45:03.366+02:00 WARN  [OTXDataAdapter] OTX API key is missing. Make sure to add the key to allow higher request limits.
2022-09-13T16:45:03.366+02:00 INFO  [LookupTableService] Data Adapter otx-api-ip/60c35d1ed7ae89169c78f333 [@cfc2c40] RUNNING
2022-09-13T16:45:03.366+02:00 INFO  [LookupTableService] Data Adapter whois/60c35d29d7ae89169c78f34c [@53f14fd] RUNNING
2022-09-13T16:45:03.368+02:00 INFO  [LookupTableService] Data Adapter otx-api-domain/60c35d1ed7ae89169c78f335 [@45f7eb77] RUNNING
2022-09-13T16:45:03.398+02:00 INFO  [LookupTableService] Cache otx-api-domain-cache/60c35d1ed7ae89169c78f336 [@5a412b19] STARTING
2022-09-13T16:45:03.403+02:00 INFO  [LookupTableService] Cache watchlist-cache/63203987d002820008fe8c91 [@313711d] STARTING
2022-09-13T16:45:03.403+02:00 INFO  [LookupTableService] Cache otx-api-domain-cache/60c35d1ed7ae89169c78f336 [@5a412b19] RUNNING
2022-09-13T16:45:03.404+02:00 INFO  [LookupTableService] Cache watchlist-cache/63203987d002820008fe8c91 [@313711d] RUNNING
2022-09-13T16:45:03.403+02:00 INFO  [LookupTableService] Cache otx-api-ip-cache/60c35d1ed7ae89169c78f331 [@3b8aff0c] STARTING
2022-09-13T16:45:03.403+02:00 INFO  [LookupTableService] Cache whois-cache/60c35d29d7ae89169c78f34a [@5b959adc] STARTING
2022-09-13T16:45:03.403+02:00 INFO  [LookupTableService] Cache dns-cache-internal/60f9dc561a6c676f0bc973bc [@50d981fc] STARTING
2022-09-13T16:45:03.404+02:00 INFO  [LookupTableService] Cache otx-api-ip-cache/60c35d1ed7ae89169c78f331 [@3b8aff0c] RUNNING
2022-09-13T16:45:03.404+02:00 INFO  [LookupTableService] Cache whois-cache/60c35d29d7ae89169c78f34a [@5b959adc] RUNNING
2022-09-13T16:45:03.405+02:00 INFO  [LookupTableService] Cache dns-cache-internal/60f9dc561a6c676f0bc973bc [@50d981fc] RUNNING
2022-09-13T16:45:03.479+02:00 INFO  [LookupTableService] Data Adapter dns-lookup-internal/60f9dc141a6c676f0bc97372 [@79e118bb] RUNNING
2022-09-13T16:45:03.484+02:00 INFO  [LookupTableService] Starting lookup table otx-api-domain/60c35d1ed7ae89169c78f338 [@19ff1954] using cache otx-api-domain-cache/60c35d1ed7ae89169c78f336 [@5a412b19], data adapter otx-api-domain/60c35d1ed7ae89169c78f335 [@45f7eb77]
2022-09-13T16:45:03.484+02:00 INFO  [LookupTableService] Starting lookup table otx-api-ip/60c35d1ed7ae89169c78f339 [@2c1536b4] using cache otx-api-ip-cache/60c35d1ed7ae89169c78f331 [@3b8aff0c], data adapter otx-api-ip/60c35d1ed7ae89169c78f333 [@cfc2c40]
2022-09-13T16:45:03.485+02:00 INFO  [LookupTableService] Starting lookup table whois/60c35d29d7ae89169c78f34d [@1f82525] using cache whois-cache/60c35d29d7ae89169c78f34a [@5b959adc], data adapter whois/60c35d29d7ae89169c78f34c [@53f14fd]
2022-09-13T16:45:03.485+02:00 INFO  [LookupTableService] Starting lookup table DNS Reverse Lookup Internal/60f9dc681a6c676f0bc973d2 [@2e14a478] using cache dns-cache-internal/60f9dc561a6c676f0bc973bc [@50d981fc], data adapter dns-lookup-internal/60f9dc141a6c676f0bc97372 [@79e118bb]
2022-09-13T16:45:03.485+02:00 INFO  [LookupTableService] Starting lookup table watchlist/63203987d002820008fe8c93 [@4f402d57] using cache watchlist-cache/63203987d002820008fe8c91 [@313711d], data adapter watchlist-mongo/63203987d002820008fe8c8f [@16ea03b4]
2022-09-13T16:45:05.999+02:00 INFO  [JerseyService] Started REST API at <0.0.0.0:9000>
2022-09-13T16:45:06.000+02:00 INFO  [ServiceManagerListener] Services are healthy
2022-09-13T16:45:06.000+02:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2022-09-13T16:45:06.003+02:00 INFO  [ServerBootstrap] Services started, startup times in ms: {FailureHandlingService [RUNNING]=1, UserSessionTerminationService [RUNNING]=1, InputSetupService [RUNNING]=3, LocalKafkaMessageQueueWriter [RUNNING]=3, GracefulShutdownService [RUNNING]=3, LocalKafkaMessageQueueReader [RUNNING]=62, UrlWhitelistService [RUNNING]=64, LocalKafkaJournal [RUNNING]=64, PrometheusExporter [RUNNING]=65, DevelopmentDirectoryObserverService [RUNNING]=65, ConfigurationEtagService [RUNNING]=66, OutputSetupService [RUNNING]=66, EtagService [RUNNING]=67, BufferSynchronizerService [RUNNING]=67, JobSchedulerService [RUNNING]=67, ProcessingConfigurationManager [RUNNING]=70, MongoDBProcessingStatusRecorderService [RUNNING]=87, PeriodicalsService [RUNNING]=92, StreamCacheService [RUNNING]=93, GeoIpDbFileChangeMonitorService [RUNNING]=102, LookupTableService [RUNNING]=241, JerseyService [RUNNING]=2759}
2022-09-13T16:45:06.005+02:00 INFO  [ServerBootstrap] Graylog server up and running.
2022-09-13T16:45:06.006+02:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2022-09-13T16:45:06.040+02:00 INFO  [InputLauncher] Not auto-starting input [Syslog TCP/Syslog TCP (port 514 forwarded by local firewall to 1514)/60c34fadd7ae89169c78e155] - desired state is STOPPED
2022-09-13T16:45:06.040+02:00 INFO  [InputLauncher] Launching input [Syslog UDP/Syslog UDP (port 514 forwarded by local firewall to 1514)/60c34fbcd7ae89169c78e169] - desired state is RUNNING
2022-09-13T16:45:06.043+02:00 INFO  [InputLauncher] Launching input [Beats/Winlogbeat/60defeba7abc3c5583c15793] - desired state is RUNNING
2022-09-13T16:45:06.046+02:00 INFO  [InputStateListener] Input [Syslog UDP/60c34fbcd7ae89169c78e169] is now STARTING
2022-09-13T16:45:06.048+02:00 INFO  [InputStateListener] Input [Beats/60defeba7abc3c5583c15793] is now STARTING
2022-09-13T16:45:06.061+02:00 INFO  [InputStateListener] Input [Beats/60defeba7abc3c5583c15793] is now RUNNING
2022-09-13T16:45:06.062+02:00 INFO  [InputStateListener] Input [Syslog UDP/60c34fbcd7ae89169c78e169] is now RUNNING
2022-09-13T16:45:06.064+02:00 WARN  [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input Beats2Input{title=Winlogbeat, type=org.graylog.plugins.beats.Beats2Input, nodeId=null} (channel [id: 0x5cd3fadb, L:/0.0.0.0:5044]) should be >= 1048576 but is 425984.
2022-09-13T16:45:06.079+02:00 WARN  [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-09-13T16:45:06.079+02:00 WARN  [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-09-13T16:45:06.079+02:00 WARN  [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-09-13T16:45:06.079+02:00 WARN  [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-09-13T16:45:06.079+02:00 WARN  [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-09-13T16:45:06.079+02:00 WARN  [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-09-13T16:45:06.079+02:00 WARN  [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-09-13T16:45:06.079+02:00 WARN  [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2022-09-13T16:45:06.082+02:00 WARN  [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file ''
2022-09-13T16:45:06.082+02:00 WARN  [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file ''
2022-09-13T16:45:06.082+02:00 WARN  [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file ''
2022-09-13T16:45:06.082+02:00 WARN  [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file ''
2022-09-13T16:45:06.082+02:00 WARN  [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file ''
2022-09-13T16:45:06.082+02:00 WARN  [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file ''
2022-09-13T16:45:06.082+02:00 WARN  [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file ''
2022-09-13T16:45:06.082+02:00 WARN  [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file ''
2022-09-13T16:45:15.294+02:00 WARN  [EnterpriseProcessingFailureHandler] No valid license found for the Enterprise Failure handler. Failures will be skipped.
2022-09-13T16:45:45.356+02:00 WARN  [EnterpriseProcessingFailureHandler] No valid license found for the Enterprise Failure handler. Failures will be skipped.

chas0rde · September 13, 2022, 2:47pm

/var/log/elasticsearch/graylog.log


[2022-09-13T16:43:55,299][INFO ][o.e.n.Node               ] [elastic] version[7.10.2], pid[23900], build[default/deb/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/5.15.0-47-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2022-09-13T16:43:55,305][INFO ][o.e.n.Node               ] [elastic] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-09-13T16:43:55,305][INFO ][o.e.n.Node               ] [elastic] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms8g, -Xmx8g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-6742182947013026966, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=4294967296, -Des.path.home=/usr/share/elasticsearch,
-Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2022-09-13T16:43:57,485][WARN ][o.e.x.w.Watcher          ] [elastic] the [action.auto_create_index] setting is configured to be restrictive [.watches,.triggered_watches,.watcher-history-*].  for the next 6 months daily history indices are allowed to be created, but please make sure that any future history indices after 6 months with the pattern [.watcher-history-yyyy.MM.dd] are allowed to be created
[2022-09-13T16:43:57,915][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [aggs-matrix-stats]
[2022-09-13T16:43:57,915][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [analysis-common]
[2022-09-13T16:43:57,916][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [constant-keyword]
[2022-09-13T16:43:57,916][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [flattened]
[2022-09-13T16:43:57,916][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [frozen-indices]
[2022-09-13T16:43:57,917][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [ingest-common]
[2022-09-13T16:43:57,917][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [ingest-geoip]
[2022-09-13T16:43:57,917][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [ingest-user-agent]
[2022-09-13T16:43:57,917][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [kibana]
[2022-09-13T16:43:57,918][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [lang-expression]
[2022-09-13T16:43:57,918][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [lang-mustache]
[2022-09-13T16:43:57,918][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [lang-painless]
[2022-09-13T16:43:57,919][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [mapper-extras]
[2022-09-13T16:43:57,919][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [mapper-version]
[2022-09-13T16:43:57,919][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [parent-join]
[2022-09-13T16:43:57,919][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [percolator]
[2022-09-13T16:43:57,920][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [rank-eval]
[2022-09-13T16:43:57,920][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [reindex]
[2022-09-13T16:43:57,920][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [repositories-metering-api]
[2022-09-13T16:43:57,921][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [repository-url]
[2022-09-13T16:43:57,921][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [search-business-rules]
[2022-09-13T16:43:57,921][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [searchable-snapshots]
[2022-09-13T16:43:57,921][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [spatial]
[2022-09-13T16:43:57,922][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [systemd]
[2022-09-13T16:43:57,922][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [transform]
[2022-09-13T16:43:57,922][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [transport-netty4]
[2022-09-13T16:43:57,922][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [unsigned-long]
[2022-09-13T16:43:57,923][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [vectors]
[2022-09-13T16:43:57,923][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [wildcard]
[2022-09-13T16:43:57,923][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-analytics]
[2022-09-13T16:43:57,924][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-async]
[2022-09-13T16:43:57,924][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-async-search]
[2022-09-13T16:43:57,924][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-autoscaling]
[2022-09-13T16:43:57,925][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-ccr]
[2022-09-13T16:43:57,925][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-core]
[2022-09-13T16:43:57,925][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-data-streams]
[2022-09-13T16:43:57,925][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-deprecation]
[2022-09-13T16:43:57,926][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-enrich]
[2022-09-13T16:43:57,926][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-eql]
[2022-09-13T16:43:57,926][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-graph]
[2022-09-13T16:43:57,927][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-identity-provider]
[2022-09-13T16:43:57,927][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-ilm]
[2022-09-13T16:43:57,927][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-logstash]
[2022-09-13T16:43:57,927][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-ml]
[2022-09-13T16:43:57,928][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-monitoring]
[2022-09-13T16:43:57,928][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-ql]
[2022-09-13T16:43:57,928][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-rollup]
[2022-09-13T16:43:57,928][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-security]
[2022-09-13T16:43:57,929][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-sql]
[2022-09-13T16:43:57,929][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-stack]
[2022-09-13T16:43:57,929][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-voting-only-node]
[2022-09-13T16:43:57,929][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-watcher]
[2022-09-13T16:43:57,930][INFO ][o.e.p.PluginsService     ] [elastic] no plugins loaded
[2022-09-13T16:43:57,969][INFO ][o.e.e.NodeEnvironment    ] [elastic] using [1] data paths, mounts [[/graylog-data (/dev/mapper/graylog--data--vg-graylog--data)]], net usable_space [409.4gb], net total_space [806gb], types [ext4]
[2022-09-13T16:43:57,969][INFO ][o.e.e.NodeEnvironment    ] [elastic] heap size [8gb], compressed ordinary object pointers [true]
[2022-09-13T16:43:58,131][INFO ][o.e.n.Node               ] [elastic] node name [elastic], node ID [jXplRySTQxW_WDlSA02XUw], cluster name [graylog], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2022-09-13T16:44:01,834][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elastic] [controller/24093] [Main.cc@114] controller (64 bit): Version 7.10.2 (Build 40a3af639d4698) Copyright (c) 2021 Elasticsearch BV
[2022-09-13T16:44:02,776][INFO ][o.e.t.NettyAllocator     ] [elastic] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2022-09-13T16:44:02,854][INFO ][o.e.d.DiscoveryModule    ] [elastic] using discovery type [single-node] and seed hosts providers [settings]
[2022-09-13T16:44:03,260][WARN ][o.e.g.DanglingIndicesState] [elastic] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-09-13T16:44:03,605][INFO ][o.e.n.Node               ] [elastic] initialized
[2022-09-13T16:44:03,607][INFO ][o.e.n.Node               ] [elastic] starting ...
[2022-09-13T16:44:03,726][INFO ][o.e.t.TransportService   ] [elastic] publish_address {10.0.221.16:9300}, bound_addresses {0.0.0.0:9300}
[2022-09-13T16:44:03,978][INFO ][o.e.c.c.Coordinator      ] [elastic] cluster UUID [04NbFYPNQXm9OtdD5yFE2w]
[2022-09-13T16:44:04,121][INFO ][o.e.c.s.MasterService    ] [elastic] elected-as-master ([1] nodes joined)[{elastic}{jXplRySTQxW_WDlSA02XUw}{FD1twWNDSSmoPh3Ahogmew}{10.0.221.16}{10.0.221.16:9300}{cdhilmrstw}{ml.machine_memory=8334688256, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 2, version: 117, delta: master node changed {previous [], current [{elastic}{jXplRySTQxW_WDlSA02XUw}{FD1twWNDSSmoPh3Ahogmew}{10.0.221.16}{10.0.221.16:9300}{cdhilmrstw}{ml.machine_memory=8334688256, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}
[2022-09-13T16:44:04,185][INFO ][o.e.c.s.ClusterApplierService] [elastic] master node changed {previous [], current [{elastic}{jXplRySTQxW_WDlSA02XUw}{FD1twWNDSSmoPh3Ahogmew}{10.0.221.16}{10.0.221.16:9300}{cdhilmrstw}{ml.machine_memory=8334688256, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 2, version: 117, reason: Publication{term=2, version=117}
[2022-09-13T16:44:04,221][INFO ][o.e.h.AbstractHttpServerTransport] [elastic] publish_address {10.0.221.16:9200}, bound_addresses {0.0.0.0:9200}
[2022-09-13T16:44:04,221][INFO ][o.e.n.Node               ] [elastic] started
[2022-09-13T16:44:04,542][INFO ][o.e.l.LicenseService     ] [elastic] license [4e52af05-bf04-4329-aaed-6ac3d1824ef9] mode [basic] - valid
[2022-09-13T16:44:04,546][INFO ][o.e.g.GatewayService     ] [elastic] recovered [5] indices into cluster_state
[2022-09-13T16:44:05,643][INFO ][o.e.c.r.a.AllocationService] [elastic] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]]]).

elastic health

 curl -X GET "localhost:9200/_cluster/health?pretty"
{
  "cluster_name" : "graylog",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 18,
  "active_shards" : 18,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Does anyone have an idea what might be wrong or where I might check why the processing isn’t working?

ihe · September 13, 2022, 2:57pm

Is there a hell of CPU load on that system and no/very slow processing? Then I would assume a runaway grok.
Java-Heap seams to be fine for Graylog, and as the output buffer is not filled I would not point in the direction of Elastic.
Did you upgrade both, Graylog and ubuntu, at the same time?
Maybe a processbuffer-dump or a thread-dump can explain what is going on. You will find it in the node overview.
grafik

chas0rde · September 14, 2022, 6:19am

Hello
high load is the java process of graylog utilizing 825%

  28576 graylog   20   0   20.0g   6.9g  21272 S 825.6  59.2   7472:03 java

How would I find a runaway grok?

Processbuffer dump is huge. What would I look for in it?
Thread dump (shortened) Part I

"kafka-journal-scheduler-1" id=331 state=TIMED_WAITING
    - waiting on <0x76ac2e25> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x76ac2e25> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-31" id=286 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-30" id=285 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-29" id=284 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-28" id=283 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"metrics-meter-tick-thread-2" id=282 state=WAITING
    - waiting on <0x2622fe96> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x2622fe96> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-27" id=281 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-26" id=279 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-25" id=277 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"metrics-meter-tick-thread-1" id=276 state=TIMED_WAITING
    - waiting on <0x2622fe96> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x2622fe96> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"kafka-journal-scheduler-0" id=275 state=WAITING
    - waiting on <0x76ac2e25> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x76ac2e25> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-24" id=271 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-23" id=267 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-22" id=266 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-21" id=265 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-20" id=264 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-19" id=263 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-18" id=262 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-17" id=261 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-16" id=259 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-15" id=258 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-14" id=257 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-13" id=256 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-12" id=255 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-11" id=254 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-10" id=252 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-9" id=250 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-8" id=248 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-7" id=247 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-6" id=246 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-5" id=245 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-29" id=243 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-4" id=242 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-28" id=241 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-27" id=240 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-3" id=239 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-26" id=238 state=TIMED_WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-25" id=237 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-24" id=236 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-23" id=235 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"http-worker-15" id=234 state=WAITING
    - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"http-worker-14" id=233 state=WAITING
    - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"http-worker-13" id=232 state=WAITING
    - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"http-worker-12" id=231 state=WAITING
    - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"http-worker-11" id=230 state=WAITING
    - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"http-worker-10" id=229 state=RUNNABLE (running in native)

    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@3d34291c
"proxied-requests-pool-2" id=228 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"http-worker-9" id=227 state=WAITING
    - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-22" id=225 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-21" id=224 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"netty-transport-1" id=223 state=RUNNABLE
    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@21f030ed
"http-worker-8" id=222 state=WAITING
    - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"http-worker-7" id=221 state=WAITING
    - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"http-worker-6" id=220 state=WAITING
    - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"http-worker-5" id=219 state=WAITING
    - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"http-worker-4" id=218 state=RUNNABLE

    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@4d32358a
"proxied-requests-pool-1" id=217 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"proxied-requests-pool-0" id=216 state=WAITING
    - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"http-worker-3" id=215 state=WAITING
    - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-20" id=214 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-19" id=213 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"SessionValidationThread-1" id=212 state=TIMED_WAITING
    - waiting on <0x42d58ad4> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x42d58ad4> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"http-worker-2" id=211 state=WAITING
    - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"http-worker-1" id=210 state=WAITING
    - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-18" id=209 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"http-worker-0" id=208 state=WAITING
    - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-17" id=207 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbuffer-processor-executor-2" id=204 state=WAITING
    - waiting on <0x26f54779> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x26f54779> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbuffer-processor-executor-2" id=203 state=WAITING
    - waiting on <0x51e5d253> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x51e5d253> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbuffer-processor-executor-2" id=202 state=WAITING
    - waiting on <0x7f48f971> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7f48f971> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbuffer-processor-executor-2" id=201 state=WAITING
    - waiting on <0x21775881> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x21775881> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbuffer-processor-executor-2" id=200 state=WAITING
    - waiting on <0x364de524> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x364de524> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbuffer-processor-executor-1" id=199 state=WAITING
    - waiting on <0x26f54779> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x26f54779> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbuffer-processor-executor-1" id=197 state=WAITING
    - waiting on <0x51e5d253> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x51e5d253> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbuffer-processor-executor-1" id=196 state=WAITING
    - waiting on <0x7f48f971> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7f48f971> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbuffer-processor-executor-1" id=195 state=WAITING
    - waiting on <0x21775881> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x21775881> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbuffer-processor-executor-1" id=194 state=WAITING
    - waiting on <0x364de524> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x364de524> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbuffer-processor-executor-0" id=193 state=WAITING
    - waiting on <0x26f54779> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x26f54779> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbuffer-processor-executor-0" id=190 state=WAITING
    - waiting on <0x7f48f971> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7f48f971> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbuffer-processor-executor-0" id=191 state=WAITING
    - waiting on <0x51e5d253> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x51e5d253> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"netty-transport-0" id=187 state=RUNNABLE
    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@3b441bd0
"outputbuffer-processor-executor-0" id=185 state=WAITING
    - waiting on <0x21775881> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x21775881> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbuffer-processor-executor-0" id=186 state=WAITING
    - waiting on <0x364de524> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x364de524> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-16" id=182 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-15" id=181 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"netty-transport-1" id=180 state=RUNNABLE
    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@2f2cd210
"netty-transport-3" id=179 state=RUNNABLE (running in native)
    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@6ed6bf86
"netty-transport-0" id=178 state=RUNNABLE
    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@7f62be6b
"netty-transport-2" id=175 state=RUNNABLE
    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@aa9f913
"netty-transport-1" id=174 state=RUNNABLE
    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@3357337a
"netty-transport-0" id=173 state=RUNNABLE
    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@6f598717
"grizzly-nio-kernel(1) SelectorRunner" id=172 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@3812c23c
"HttpServer-0" id=171 state=TIMED_WAITING
    - waiting on <0x651e566a> (a java.lang.Object)
    - locked <0x651e566a> (a java.lang.Object)
    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@9222702
"scheduled-14" id=170 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-13" id=169 state=TIMED_WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-12" id=168 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-11" id=167 state=TIMED_WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"nioEventLoopGroup-2-1" id=166 state=RUNNABLE (running in native)
      - locked sun.nio.ch.EPollSelectorImpl@7b9ea9eb
"scheduled-10" id=165 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-9" id=154 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-8" id=153 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-29" id=151 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-28" id=150 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-27" id=149 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-26" id=148 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-25" id=147 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-7" id=146 state=TIMED_WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-24" id=145 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-23" id=144 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-22" id=142 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-21" id=141 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-20" id=140 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-19" id=139 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-18" id=138 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-17" id=137 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-16" id=136 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-15" id=135 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"job-heartbeat-0" id=134 state=TIMED_WAITING
    - waiting on <0x6798a455> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x6798a455> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-14" id=133 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-13" id=131 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-12" id=130 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-11" id=129 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-10" id=128 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-9" id=127 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-8" id=126 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-6" id=125 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-7" id=124 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-6" id=123 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-5" id=122 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-5" id=119 state=TIMED_WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-4" id=116 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"JobSchedulerService" id=113 state=TIMED_WAITING
    - waiting on <0x4aea8e9c> (a java.util.concurrent.Semaphore$NonfairSync)
    - locked <0x4aea8e9c> (a java.util.concurrent.Semaphore$NonfairSync)

Seems most stuff is waiting (though it seems according to the CPU load maybe busy-waiting)

Release upgrade was done on both systems on the same day

chas0rde · September 14, 2022, 6:19am

Part II

"scheduled-4" id=112 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"output-setup-service-0" id=108 state=WAITING
    - waiting on <0x523d8b41> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x523d8b41> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-3" id=107 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-2" id=105 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-1" id=103 state=TIMED_WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-3" id=102 state=TIMED_WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-0" id=100 state=WAITING
    - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"LocalKafkaMessageQueueReader" id=95 state=TIMED_WAITING
"InputSetupService" id=90 state=WAITING
    - waiting on <0x483cbc68> (a java.util.concurrent.CountDownLatch$Sync)
    - locked <0x483cbc68> (a java.util.concurrent.CountDownLatch$Sync)
"FailureHandlingService" id=88 state=WAITING
    - waiting on <0x504900ca> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x504900ca> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"cluster-eventbus-handler-1" id=85 state=WAITING
    - waiting on <0x619dd25a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x619dd25a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"eventbus-handler-1" id=84 state=WAITING
    - waiting on <0x7f6d5f3e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7f6d5f3e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"eventbus-handler-0" id=83 state=WAITING
    - waiting on <0x7f6d5f3e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7f6d5f3e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"cluster-eventbus-handler-0" id=82 state=WAITING
    - waiting on <0x619dd25a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x619dd25a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-2" id=81 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-1" id=80 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"processbufferprocessor-7" id=79 state=RUNNABLE
"processbufferprocessor-6" id=78 state=RUNNABLE
"processbufferprocessor-5" id=77 state=RUNNABLE
"processbufferprocessor-4" id=76 state=RUNNABLE
"processbufferprocessor-3" id=75 state=RUNNABLE
"processbufferprocessor-2" id=74 state=RUNNABLE
"processbufferprocessor-1" id=73 state=RUNNABLE
"processbufferprocessor-0" id=72 state=RUNNABLE
"aws-instance-lookup-refresher-0" id=71 state=TIMED_WAITING
    - waiting on <0x0ab24be0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x0ab24be0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"aws-instance-lookup-refresher-0" id=70 state=TIMED_WAITING
    - waiting on <0x21d1db52> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x21d1db52> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"aws-instance-lookup-refresher-0" id=69 state=TIMED_WAITING
    - waiting on <0x27173f28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x27173f28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"aws-instance-lookup-refresher-0" id=68 state=TIMED_WAITING
    - waiting on <0x6521dcb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x6521dcb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"aws-instance-lookup-refresher-0" id=67 state=TIMED_WAITING
    - waiting on <0x58ae6cc4> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x58ae6cc4> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"aws-instance-lookup-refresher-0" id=66 state=TIMED_WAITING
    - waiting on <0x2d4570a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x2d4570a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"aws-instance-lookup-refresher-0" id=65 state=TIMED_WAITING
    - waiting on <0x0be933b7> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x0be933b7> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbufferprocessor-4" id=64 state=WAITING
    - waiting on <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbufferprocessor-3" id=63 state=WAITING
    - waiting on <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbufferprocessor-2" id=62 state=WAITING
    - waiting on <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbufferprocessor-1" id=61 state=WAITING
    - waiting on <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"outputbufferprocessor-0" id=60 state=WAITING
    - waiting on <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"aws-instance-lookup-refresher-0" id=59 state=TIMED_WAITING
    - waiting on <0x75e01fc3> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x75e01fc3> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"scheduled-daemon-0" id=58 state=WAITING
    - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"I/O dispatcher 20" id=57 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@33897c3c
"I/O dispatcher 19" id=56 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@65ecac
"I/O dispatcher 18" id=55 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@5a98a565
"I/O dispatcher 17" id=54 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@2fb703e
"I/O dispatcher 16" id=53 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@77854aaa
"I/O dispatcher 15" id=52 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@1d39f74e
"I/O dispatcher 14" id=51 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@545e368f
"I/O dispatcher 13" id=50 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@5771c1a6
"I/O dispatcher 12" id=49 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@5456fe8d
"I/O dispatcher 11" id=48 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@2b9f916b
"I/O dispatcher 10" id=47 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@7fcc505b
"I/O dispatcher 9" id=46 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@66b50a4a
"I/O dispatcher 8" id=45 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@6cafb46e
"I/O dispatcher 7" id=44 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@a95537a
"I/O dispatcher 6" id=43 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@2ecff4f0
"I/O dispatcher 5" id=42 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@6c822e24
"I/O dispatcher 4" id=41 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@1eadf5c3
"I/O dispatcher 3" id=40 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@2b08316e
"I/O dispatcher 2" id=39 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@4ef144ab
"I/O dispatcher 1" id=38 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@4d81a6eb
"pool-9-thread-1" id=37 state=RUNNABLE
      - locked sun.nio.ch.EPollSelectorImpl@c95e691
"OkHttp ConnectionPool" id=36 state=TIMED_WAITING
    - waiting on <0x7412985f> (a okhttp3.internal.connection.RealConnectionPool)
    - locked <0x7412985f> (a okhttp3.internal.connection.RealConnectionPool)
    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@269a37bc
"inputbufferprocessor-2" id=35 state=WAITING
    - waiting on <0x6a897190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x6a897190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"inputbufferprocessor-1" id=34 state=WAITING
    - waiting on <0x6a897190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x6a897190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"inputbufferprocessor-0" id=33 state=WAITING
    - waiting on <0x6a897190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x6a897190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"CleanCursors-2-thread-1" id=32 state=TIMED_WAITING
    - waiting on <0x3e3a695b> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x3e3a695b> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"cluster-ClusterId{value='6320976d2ff9f01e48bb4ff6', description='null'}-localhost:27017" id=31 state=TIMED_WAITING
    - waiting on <0x45da0aa5> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    - locked <0x45da0aa5> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
"Okio Watchdog" id=28 state=TIMED_WAITING
    - waiting on <0x10b4e7f8> (a java.lang.Class)
    - locked <0x10b4e7f8> (a java.lang.Class)
"Common-Cleaner" id=19 state=TIMED_WAITING
    - waiting on <0x562058f4> (a java.lang.ref.ReferenceQueue$Lock)
    - locked <0x562058f4> (a java.lang.ref.ReferenceQueue$Lock)
"Signal Dispatcher" id=4 state=RUNNABLE
"Finalizer" id=3 state=WAITING
    - waiting on <0x095e58b5> (a java.lang.ref.ReferenceQueue$Lock)
    - locked <0x095e58b5> (a java.lang.ref.ReferenceQueue$Lock)
"Reference Handler" id=2 state=RUNNABLE
"main" id=1 state=WAITING
    - waiting on <0x79d743e6> (a java.lang.Thread)
    - locked <0x79d743e6> (a java.lang.Thread)

gsmith · September 19, 2022, 9:48pm

Hello,
Do you have any pipelines or extractors on this Graylog Server? If so, for troubleshooting I would looking into disabling or remove them to see if it clears up.

system · October 3, 2022, 9:48pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog not processing data Graylog Central (peer support)	13	6933	March 30, 2017
Struggling with Graylog stopping to export to Elasticsearch Graylog Central (peer support) pipeline-rules , debuggingpl	14	2019	August 5, 2021
Performace Problems Graylog Central (peer support)	5	560	December 6, 2019
Graylog doesn't process messages anymore, ELS cluster is green Graylog Central (peer support)	6	3538	April 12, 2021
Elasticsearch optimization Graylog Central (peer support)	3	1027	January 23, 2023

Graylog not processing to Elasticseach anymore

Related topics