# Graylog not processing to Elasticseach anymore **URL:** https://community.graylog.org/t/graylog-not-processing-to-elasticseach-anymore/25700 **Category:** Graylog Central (peer support) **Created:** [September 13, 2022, 2:46pm UTC](https://community.graylog.org/t/graylog-not-processing-to-elasticseach-anymore/25700 "2022-09-13T14:46:46Z") **Posts on this page:** 8 **Page:** 1
### Author: ![chas0rde](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/chas0rde/32/9035_2.png) [@chas0rde](https://community.graylog.org/u/chas0rde) #### Post date: [September 13, 2022, 2:46pm UTC](https://community.graylog.org/t/graylog-not-processing-to-elasticseach-anymore/25700/1 "2022-09-13T14:46:46Z")
Out graylog instance stopped processing after ubuntu release upgrade from 20.04LTS to 22.04LTS From my point of view versions are ok. We have two systems (1.Graylog+MongoDB and 2. Elasticsearch) Graylog is Graylog 4.3.6+36120cf (behind apache2 as https proxy) Elasticsearch is 7.10.2 Processbuffer is full and Journal keeps growing: ![grafik](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/f/fa7a9374a34584f631507b2e28891ea02b297904.png) **/etc/graylog/server/server.conf** (comments and commented-out parameters removed) ```auto is_master = true node_id_file = /etc/graylog/server/node-id password_secret = root_password_sha2 = bin_dir = /usr/share/graylog-server/bin data_dir = /var/lib/graylog-server plugin_dir = /usr/share/graylog-server/plugin http_bind_address = 0.0.0.0:9000 http_external_uri = https://graylog.monitoring.somedomain.local/ elasticsearch_hosts = http://elastic:9200 rotation_strategy = count elasticsearch_max_docs_per_index = 20000000 elasticsearch_max_number_of_indices = 20 retention_strategy = delete elasticsearch_shards = 4 elasticsearch_replicas = 0 elasticsearch_index_prefix = graylog allow_leading_wildcard_searches = true allow_highlighting = false elasticsearch_analyzer = standard output_batch_size = 500 output_flush_interval = 1 output_fault_count_threshold = 5 output_fault_penalty_seconds = 30 processbuffer_processors = 8 outputbuffer_processors = 5 processor_wait_strategy = blocking ring_size = 65536 inputbuffer_ring_size = 65536 inputbuffer_processors = 2 inputbuffer_wait_strategy = blocking message_journal_enabled = true message_journal_dir = /var/lib/graylog-server/journal message_journal_max_age = 48h message_journal_max_size = 20gb lb_recognition_period_seconds = 3 mongodb_uri = mongodb://localhost/graylog mongodb_max_connections = 1000 mongodb_threads_allowed_to_block_multiplier = 5 proxied_requests_thread_pool_size = 32 elasticsearch_mute_deprecation_warnings=false ``` **/etc/elasticsearch/elasticsearch.yml** ```auto cluster.name: graylog path.data: /graylog-data/elasticsearch path.logs: /var/log/elasticsearch network.host: 0.0.0.0 discovery.type: single-node xpack.security.enabled: false action.auto_create_index: .watches,.triggered_watches,.watcher-history-* ``` ---
### Author: ![chas0rde](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/chas0rde/32/9035_2.png) [@chas0rde](https://community.graylog.org/u/chas0rde) #### Post date: [September 13, 2022, 2:47pm UTC](https://community.graylog.org/t/graylog-not-processing-to-elasticseach-anymore/25700/2 "2022-09-13T14:47:14Z")
Some logs of both services starting. Elasticsearch shows health to be green (see log below) **/var/log/graylog-server/server.log** ```auto 2022-09-13T16:44:57.497+02:00 INFO [ImmutableFeatureFlagsCollector] Following feature flags are used: {} 2022-09-13T16:44:58.029+02:00 INFO [CmdLineTool] Loaded plugin: AWS plugins 4.3.6 [org.graylog.aws.AWSPlugin] 2022-09-13T16:44:58.029+02:00 INFO [CmdLineTool] Loaded plugin: Enterprise Integrations 4.3.6 [org.graylog.enterprise.integrations.EnterpriseIntegrationsPlugin] 2022-09-13T16:44:58.030+02:00 INFO [CmdLineTool] Loaded plugin: Integrations 4.3.6 [org.graylog.integrations.IntegrationsPlugin] 2022-09-13T16:44:58.031+02:00 INFO [CmdLineTool] Loaded plugin: Collector 4.3.6 [org.graylog.plugins.collector.CollectorPlugin] 2022-09-13T16:44:58.031+02:00 INFO [CmdLineTool] Loaded plugin: Graylog Enterprise 4.3.6 [org.graylog.plugins.enterprise.EnterprisePlugin] 2022-09-13T16:44:58.032+02:00 INFO [CmdLineTool] Loaded plugin: Graylog Enterprise (ES6 Support) 4.3.6 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es6.EnterpriseES6Plugin] 2022-09-13T16:44:58.032+02:00 INFO [CmdLineTool] Loaded plugin: Graylog Enterprise (ES7 Support) 4.3.6 [org.graylog.plugins.enterprise.org.graylog.plugins.enterprise.es7.EnterpriseES7Plugin] 2022-09-13T16:44:58.033+02:00 INFO [CmdLineTool] Loaded plugin: Threat Intelligence Plugin 4.3.6 [org.graylog.plugins.threatintel.ThreatIntelPlugin] 2022-09-13T16:44:58.033+02:00 INFO [CmdLineTool] Loaded plugin: Elasticsearch 6 Support 4.3.6+36120cf [org.graylog.storage.elasticsearch6.Elasticsearch6Plugin] 2022-09-13T16:44:58.033+02:00 INFO [CmdLineTool] Loaded plugin: Elasticsearch 7 Support 4.3.6+36120cf [org.graylog.storage.elasticsearch7.Elasticsearch7Plugin] 2022-09-13T16:44:58.055+02:00 INFO [CmdLineTool] Running with JVM arguments: -Xms8g -Xmx8g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djdk.tls.acknowledgeCloseNotify=true -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb 2022-09-13T16:44:58.312+02:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000} 2022-09-13T16:44:58.338+02:00 INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out 2022-09-13T16:44:58.350+02:00 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:109}] to localhost:27017 2022-09-13T16:44:58.353+02:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 16]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=2006759} 2022-09-13T16:44:58.362+02:00 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:110}] to localhost:27017 2022-09-13T16:44:58.378+02:00 INFO [connection] Closed connection [connectionId{localValue:2, serverValue:110}] to localhost:27017 because the pool has been closed. 2022-09-13T16:44:58.380+02:00 INFO [MongoDBPreflightCheck] Connected to MongoDB version 4.4.16 2022-09-13T16:44:58.547+02:00 INFO [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version 2022-09-13T16:44:58.642+02:00 INFO [Version] HV000001: Hibernate Validator null 2022-09-13T16:45:00.734+02:00 INFO [InputBufferImpl] Message journal is enabled. 2022-09-13T16:45:00.746+02:00 INFO [NodeId] Node ID: 327f2801-d3ee-4215-bfe3-36016b482b56 2022-09-13T16:45:00.868+02:00 INFO [LogManager] Loading logs. 2022-09-13T16:45:01.063+02:00 INFO [LogManager] Logs loading complete. 2022-09-13T16:45:01.065+02:00 INFO [LocalKafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal 2022-09-13T16:45:01.069+02:00 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000} 2022-09-13T16:45:01.071+02:00 INFO [cluster] Cluster description not yet available. Waiting for 30000 ms before timing out 2022-09-13T16:45:01.072+02:00 INFO [connection] Opened connection [connectionId{localValue:3, serverValue:111}] to localhost:27017 2022-09-13T16:45:01.073+02:00 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 4, 16]}, minWireVersion=0, maxWireVersion=9, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=529289} 2022-09-13T16:45:01.080+02:00 INFO [connection] Opened connection [connectionId{localValue:4, serverValue:112}] to localhost:27017 2022-09-13T16:45:01.188+02:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers. 2022-09-13T16:45:01.308+02:00 INFO [ElasticsearchVersionProvider] Elasticsearch cluster is running Elasticsearch:7.10.2 2022-09-13T16:45:01.895+02:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy . 2022-09-13T16:45:02.087+02:00 INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy . 2022-09-13T16:45:02.149+02:00 INFO [connection] Opened connection [connectionId{localValue:5, serverValue:113}] to localhost:27017 2022-09-13T16:45:02.676+02:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license. 2022-09-13T16:45:02.677+02:00 INFO [ServerBootstrap] Graylog server 4.3.6+36120cf starting up 2022-09-13T16:45:02.678+02:00 INFO [ServerBootstrap] JRE: Ubuntu 11.0.16 on Linux 5.15.0-47-generic 2022-09-13T16:45:02.678+02:00 INFO [ServerBootstrap] Deployment: deb 2022-09-13T16:45:02.678+02:00 INFO [ServerBootstrap] OS: Ubuntu 22.04.1 LTS (jammy) 2022-09-13T16:45:02.678+02:00 INFO [ServerBootstrap] Arch: amd64 2022-09-13T16:45:02.769+02:00 INFO [ServerBootstrap] Running 62 migrations... 2022-09-13T16:45:03.242+02:00 INFO [PeriodicalsService] Starting 37 periodicals ... 2022-09-13T16:45:03.245+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s]. 2022-09-13T16:45:03.246+02:00 INFO [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever. 2022-09-13T16:45:03.247+02:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.AlertScannerThread] periodical. Not configured to run on this node. 2022-09-13T16:45:03.247+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s]. 2022-09-13T16:45:03.247+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s]. 2022-09-13T16:45:03.248+02:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical. Not configured to run on this node. 2022-09-13T16:45:03.248+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever. 2022-09-13T16:45:03.249+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s]. 2022-09-13T16:45:03.249+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexBlockCheck] periodical in [0s], polling every [30s]. 2022-09-13T16:45:03.250+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s]. 2022-09-13T16:45:03.250+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s]. 2022-09-13T16:45:03.251+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s]. 2022-09-13T16:45:03.251+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s]. 2022-09-13T16:45:03.251+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s]. 2022-09-13T16:45:03.252+02:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s]. 2022-09-13T16:45:03.253+02:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s]. 2022-09-13T16:45:03.303+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever. 2022-09-13T16:45:03.303+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s]. 2022-09-13T16:45:03.304+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.TrafficCounterCalculator] periodical in [0s], polling every [1s]. 2022-09-13T16:45:03.306+02:00 INFO [Periodicals] Starting [org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical] periodical in [0s], polling every [1s]. 2022-09-13T16:45:03.305+02:00 INFO [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed. 2022-09-13T16:45:03.308+02:00 INFO [connection] Opened connection [connectionId{localValue:6, serverValue:114}] to localhost:27017 2022-09-13T16:45:03.308+02:00 INFO [connection] Opened connection [connectionId{localValue:7, serverValue:115}] to localhost:27017 2022-09-13T16:45:03.307+02:00 INFO [connection] Opened connection [connectionId{localValue:8, serverValue:116}] to localhost:27017 2022-09-13T16:45:03.308+02:00 INFO [connection] Opened connection [connectionId{localValue:9, serverValue:117}] to localhost:27017 2022-09-13T16:45:03.309+02:00 INFO [Periodicals] Starting [org.graylog.scheduler.periodicals.ScheduleTriggerCleanUp] periodical in [120s], polling every [86400s]. 2022-09-13T16:45:03.310+02:00 INFO [connection] Opened connection [connectionId{localValue:11, serverValue:118}] to localhost:27017 2022-09-13T16:45:03.310+02:00 INFO [connection] Opened connection [connectionId{localValue:10, serverValue:119}] to localhost:27017 2022-09-13T16:45:03.310+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.ESVersionCheckPeriodical] periodical in [0s], polling every [30s]. 2022-09-13T16:45:03.313+02:00 INFO [Periodicals] Starting [org.graylog2.periodical.UserSessionTerminationPeriodical] periodical, running forever. 2022-09-13T16:45:03.315+02:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredSidecarsThread] periodical in [0s], polling every [600s]. 2022-09-13T16:45:03.316+02:00 INFO [Periodicals] Starting [org.graylog.plugins.sidecar.periodical.PurgeExpiredConfigurationUploads] periodical in [0s], polling every [600s]. 2022-09-13T16:45:03.319+02:00 INFO [Periodicals] Starting [org.graylog.plugins.views.search.db.SearchesCleanUpJob] periodical in [3600s], polling every [28800s]. 2022-09-13T16:45:03.319+02:00 INFO [Periodicals] Starting [org.graylog.events.periodicals.EventNotificationStatusCleanUp] periodical in [120s], polling every [86400s]. 2022-09-13T16:45:03.320+02:00 INFO [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s]. 2022-09-13T16:45:03.321+02:00 INFO [Periodicals] Starting [org.graylog.plugins.forwarder.ForwarderStatePeriodical] periodical in [0s], polling every [15s]. 2022-09-13T16:45:03.324+02:00 INFO [Periodicals] Starting [org.graylog.plugins.license.LicenseManagerPeriodical] periodical in [0s], polling every [300s]. 2022-09-13T16:45:03.328+02:00 INFO [Periodicals] Starting [org.graylog.plugins.license.LicenseReportPeriodical] periodical in [300s], polling every [3600s]. 2022-09-13T16:45:03.328+02:00 INFO [Periodicals] Starting [org.graylog.plugins.license.StagedLicenseInstallerPeriodical] periodical, running forever. 2022-09-13T16:45:03.328+02:00 INFO [Periodicals] Starting [org.graylog.plugins.auditlog.mongodb.MongoAuditLogPeriodical] periodical in [0s], polling every [3600s]. 2022-09-13T16:45:03.329+02:00 ERROR [MongoAuditLogPeriodical] Not running cleanup for auditlog entries in MongoDB because there is no valid license. 2022-09-13T16:45:03.329+02:00 INFO [Periodicals] Starting [org.graylog.plugins.report.scheduler.ReportPeriodical] periodical in [120s], polling every [60s]. 2022-09-13T16:45:03.329+02:00 INFO [Periodicals] Starting [org.graylog.plugins.files.CleanupPeriodical] periodical in [0s], polling every [86400s]. 2022-09-13T16:45:03.330+02:00 INFO [PeriodicalsService] Not starting [org.graylog.plugins.securityapp.anomaly.retrieval.AnomalyRetrievalPeriodical] periodical. Not configured to run on this node. 2022-09-13T16:45:03.330+02:00 INFO [PeriodicalsService] Not starting [org.graylog.plugins.securityapp.anomaly.DetectorStatusSyncPeriodical] periodical. Not configured to run on this node. 2022-09-13T16:45:03.338+02:00 WARN [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb 2022-09-13T16:45:03.341+02:00 ERROR [GeoIpDbFileChangeMonitorService] Error validating GeoIP Database files. Invalid 'MAXMIND' City Geo IP database file '/etc/graylog/server/GeoLite2-City.mmdb'. Make sure the file exists and is valid for 'MAXMIND' org.graylog2.plugin.validate.ConfigValidationException: Invalid 'MAXMIND' City Geo IP database file '/etc/graylog/server/GeoLite2-City.mmdb'. Make sure the file exists and is valid for 'MAXMIND' at org.graylog2.rest.resources.system.GeoIpResolverConfigValidator.validateConfig(GeoIpResolverConfigValidator.java:93) ~[graylog.jar:?] at org.graylog2.rest.resources.system.GeoIpResolverConfigValidator.validate(GeoIpResolverConfigValidator.java:65) ~[graylog.jar:?] at org.graylog.plugins.map.geoip.GeoIpDbFileChangeMonitorService.updateConfiguration(GeoIpDbFileChangeMonitorService.java:147) [graylog.jar:?] at org.graylog.plugins.map.geoip.GeoIpDbFileChangeMonitorService.startUp(GeoIpDbFileChangeMonitorService.java:97) [graylog.jar:?] at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) [graylog.jar:?] at com.google.common.util.concurrent.Callables$4.run(Callables.java:121) [graylog.jar:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2022-09-13T16:45:03.364+02:00 INFO [LookupTableService] Data Adapter watchlist-mongo/63203987d002820008fe8c8f [@16ea03b4] STARTING 2022-09-13T16:45:03.364+02:00 INFO [LookupTableService] Data Adapter dns-lookup-internal/60f9dc141a6c676f0bc97372 [@79e118bb] STARTING 2022-09-13T16:45:03.364+02:00 INFO [LookupTableService] Data Adapter otx-api-ip/60c35d1ed7ae89169c78f333 [@cfc2c40] STARTING 2022-09-13T16:45:03.364+02:00 INFO [LookupTableService] Data Adapter whois/60c35d29d7ae89169c78f34c [@53f14fd] STARTING 2022-09-13T16:45:03.364+02:00 WARN [OTXDataAdapter] OTX API key is missing. Make sure to add the key to allow higher request limits. 2022-09-13T16:45:03.366+02:00 INFO [LookupTableService] Data Adapter otx-api-domain/60c35d1ed7ae89169c78f335 [@45f7eb77] STARTING 2022-09-13T16:45:03.366+02:00 INFO [LookupTableService] Data Adapter watchlist-mongo/63203987d002820008fe8c8f [@16ea03b4] RUNNING 2022-09-13T16:45:03.366+02:00 WARN [OTXDataAdapter] OTX API key is missing. Make sure to add the key to allow higher request limits. 2022-09-13T16:45:03.366+02:00 INFO [LookupTableService] Data Adapter otx-api-ip/60c35d1ed7ae89169c78f333 [@cfc2c40] RUNNING 2022-09-13T16:45:03.366+02:00 INFO [LookupTableService] Data Adapter whois/60c35d29d7ae89169c78f34c [@53f14fd] RUNNING 2022-09-13T16:45:03.368+02:00 INFO [LookupTableService] Data Adapter otx-api-domain/60c35d1ed7ae89169c78f335 [@45f7eb77] RUNNING 2022-09-13T16:45:03.398+02:00 INFO [LookupTableService] Cache otx-api-domain-cache/60c35d1ed7ae89169c78f336 [@5a412b19] STARTING 2022-09-13T16:45:03.403+02:00 INFO [LookupTableService] Cache watchlist-cache/63203987d002820008fe8c91 [@313711d] STARTING 2022-09-13T16:45:03.403+02:00 INFO [LookupTableService] Cache otx-api-domain-cache/60c35d1ed7ae89169c78f336 [@5a412b19] RUNNING 2022-09-13T16:45:03.404+02:00 INFO [LookupTableService] Cache watchlist-cache/63203987d002820008fe8c91 [@313711d] RUNNING 2022-09-13T16:45:03.403+02:00 INFO [LookupTableService] Cache otx-api-ip-cache/60c35d1ed7ae89169c78f331 [@3b8aff0c] STARTING 2022-09-13T16:45:03.403+02:00 INFO [LookupTableService] Cache whois-cache/60c35d29d7ae89169c78f34a [@5b959adc] STARTING 2022-09-13T16:45:03.403+02:00 INFO [LookupTableService] Cache dns-cache-internal/60f9dc561a6c676f0bc973bc [@50d981fc] STARTING 2022-09-13T16:45:03.404+02:00 INFO [LookupTableService] Cache otx-api-ip-cache/60c35d1ed7ae89169c78f331 [@3b8aff0c] RUNNING 2022-09-13T16:45:03.404+02:00 INFO [LookupTableService] Cache whois-cache/60c35d29d7ae89169c78f34a [@5b959adc] RUNNING 2022-09-13T16:45:03.405+02:00 INFO [LookupTableService] Cache dns-cache-internal/60f9dc561a6c676f0bc973bc [@50d981fc] RUNNING 2022-09-13T16:45:03.479+02:00 INFO [LookupTableService] Data Adapter dns-lookup-internal/60f9dc141a6c676f0bc97372 [@79e118bb] RUNNING 2022-09-13T16:45:03.484+02:00 INFO [LookupTableService] Starting lookup table otx-api-domain/60c35d1ed7ae89169c78f338 [@19ff1954] using cache otx-api-domain-cache/60c35d1ed7ae89169c78f336 [@5a412b19], data adapter otx-api-domain/60c35d1ed7ae89169c78f335 [@45f7eb77] 2022-09-13T16:45:03.484+02:00 INFO [LookupTableService] Starting lookup table otx-api-ip/60c35d1ed7ae89169c78f339 [@2c1536b4] using cache otx-api-ip-cache/60c35d1ed7ae89169c78f331 [@3b8aff0c], data adapter otx-api-ip/60c35d1ed7ae89169c78f333 [@cfc2c40] 2022-09-13T16:45:03.485+02:00 INFO [LookupTableService] Starting lookup table whois/60c35d29d7ae89169c78f34d [@1f82525] using cache whois-cache/60c35d29d7ae89169c78f34a [@5b959adc], data adapter whois/60c35d29d7ae89169c78f34c [@53f14fd] 2022-09-13T16:45:03.485+02:00 INFO [LookupTableService] Starting lookup table DNS Reverse Lookup Internal/60f9dc681a6c676f0bc973d2 [@2e14a478] using cache dns-cache-internal/60f9dc561a6c676f0bc973bc [@50d981fc], data adapter dns-lookup-internal/60f9dc141a6c676f0bc97372 [@79e118bb] 2022-09-13T16:45:03.485+02:00 INFO [LookupTableService] Starting lookup table watchlist/63203987d002820008fe8c93 [@4f402d57] using cache watchlist-cache/63203987d002820008fe8c91 [@313711d], data adapter watchlist-mongo/63203987d002820008fe8c8f [@16ea03b4] 2022-09-13T16:45:05.999+02:00 INFO [JerseyService] Started REST API at <0.0.0.0:9000> 2022-09-13T16:45:06.000+02:00 INFO [ServiceManagerListener] Services are healthy 2022-09-13T16:45:06.000+02:00 INFO [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE] 2022-09-13T16:45:06.003+02:00 INFO [ServerBootstrap] Services started, startup times in ms: {FailureHandlingService [RUNNING]=1, UserSessionTerminationService [RUNNING]=1, InputSetupService [RUNNING]=3, LocalKafkaMessageQueueWriter [RUNNING]=3, GracefulShutdownService [RUNNING]=3, LocalKafkaMessageQueueReader [RUNNING]=62, UrlWhitelistService [RUNNING]=64, LocalKafkaJournal [RUNNING]=64, PrometheusExporter [RUNNING]=65, DevelopmentDirectoryObserverService [RUNNING]=65, ConfigurationEtagService [RUNNING]=66, OutputSetupService [RUNNING]=66, EtagService [RUNNING]=67, BufferSynchronizerService [RUNNING]=67, JobSchedulerService [RUNNING]=67, ProcessingConfigurationManager [RUNNING]=70, MongoDBProcessingStatusRecorderService [RUNNING]=87, PeriodicalsService [RUNNING]=92, StreamCacheService [RUNNING]=93, GeoIpDbFileChangeMonitorService [RUNNING]=102, LookupTableService [RUNNING]=241, JerseyService [RUNNING]=2759} 2022-09-13T16:45:06.005+02:00 INFO [ServerBootstrap] Graylog server up and running. 2022-09-13T16:45:06.006+02:00 ERROR [AuditLogger] Unable to write audit log entry because there is no valid license. 2022-09-13T16:45:06.040+02:00 INFO [InputLauncher] Not auto-starting input [Syslog TCP/Syslog TCP (port 514 forwarded by local firewall to 1514)/60c34fadd7ae89169c78e155] - desired state is STOPPED 2022-09-13T16:45:06.040+02:00 INFO [InputLauncher] Launching input [Syslog UDP/Syslog UDP (port 514 forwarded by local firewall to 1514)/60c34fbcd7ae89169c78e169] - desired state is RUNNING 2022-09-13T16:45:06.043+02:00 INFO [InputLauncher] Launching input [Beats/Winlogbeat/60defeba7abc3c5583c15793] - desired state is RUNNING 2022-09-13T16:45:06.046+02:00 INFO [InputStateListener] Input [Syslog UDP/60c34fbcd7ae89169c78e169] is now STARTING 2022-09-13T16:45:06.048+02:00 INFO [InputStateListener] Input [Beats/60defeba7abc3c5583c15793] is now STARTING 2022-09-13T16:45:06.061+02:00 INFO [InputStateListener] Input [Beats/60defeba7abc3c5583c15793] is now RUNNING 2022-09-13T16:45:06.062+02:00 INFO [InputStateListener] Input [Syslog UDP/60c34fbcd7ae89169c78e169] is now RUNNING 2022-09-13T16:45:06.064+02:00 WARN [AbstractTcpTransport] receiveBufferSize (SO_RCVBUF) for input Beats2Input{title=Winlogbeat, type=org.graylog.plugins.beats.Beats2Input, nodeId=null} (channel [id: 0x5cd3fadb, L:/0.0.0.0:5044]) should be >= 1048576 but is 425984. 2022-09-13T16:45:06.079+02:00 WARN [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb 2022-09-13T16:45:06.079+02:00 WARN [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb 2022-09-13T16:45:06.079+02:00 WARN [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb 2022-09-13T16:45:06.079+02:00 WARN [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb 2022-09-13T16:45:06.079+02:00 WARN [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb 2022-09-13T16:45:06.079+02:00 WARN [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb 2022-09-13T16:45:06.079+02:00 WARN [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb 2022-09-13T16:45:06.079+02:00 WARN [GeoIpResolver] 'org.graylog.plugins.map.geoip.MaxMindIpLocationResolver' database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb 2022-09-13T16:45:06.082+02:00 WARN [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file '' 2022-09-13T16:45:06.082+02:00 WARN [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file '' 2022-09-13T16:45:06.082+02:00 WARN [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file '' 2022-09-13T16:45:06.082+02:00 WARN [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file '' 2022-09-13T16:45:06.082+02:00 WARN [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file '' 2022-09-13T16:45:06.082+02:00 WARN [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file '' 2022-09-13T16:45:06.082+02:00 WARN [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file '' 2022-09-13T16:45:06.082+02:00 WARN [MaxMindIpResolver] Error creating DatabaseReader for 'MaxMindIpAsnResolver' with config file '' 2022-09-13T16:45:15.294+02:00 WARN [EnterpriseProcessingFailureHandler] No valid license found for the Enterprise Failure handler. Failures will be skipped. 2022-09-13T16:45:45.356+02:00 WARN [EnterpriseProcessingFailureHandler] No valid license found for the Enterprise Failure handler. Failures will be skipped. ``` ---
### Author: ![chas0rde](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/chas0rde/32/9035_2.png) [@chas0rde](https://community.graylog.org/u/chas0rde) #### Post date: [September 13, 2022, 2:47pm UTC](https://community.graylog.org/t/graylog-not-processing-to-elasticseach-anymore/25700/3 "2022-09-13T14:47:19Z")
**/var/log/elasticsearch/graylog.log** ```auto [2022-09-13T16:43:55,299][INFO][o.e.n.Node] [elastic] version[7.10.2], pid[23900], build[default/deb/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/5.15.0-47-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9] [2022-09-13T16:43:55,305][INFO][o.e.n.Node] [elastic] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true] [2022-09-13T16:43:55,305][INFO][o.e.n.Node] [elastic] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms8g, -Xmx8g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-6742182947013026966, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=4294967296, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true] [2022-09-13T16:43:57,485][WARN][o.e.x.w.Watcher] [elastic] the [action.auto_create_index] setting is configured to be restrictive [.watches,.triggered_watches,.watcher-history-*]. for the next 6 months daily history indices are allowed to be created, but please make sure that any future history indices after 6 months with the pattern [.watcher-history-yyyy.MM.dd] are allowed to be created [2022-09-13T16:43:57,915][INFO][o.e.p.PluginsService] [elastic] loaded module [aggs-matrix-stats] [2022-09-13T16:43:57,915][INFO][o.e.p.PluginsService] [elastic] loaded module [analysis-common] [2022-09-13T16:43:57,916][INFO][o.e.p.PluginsService] [elastic] loaded module [constant-keyword] [2022-09-13T16:43:57,916][INFO][o.e.p.PluginsService] [elastic] loaded module [flattened] [2022-09-13T16:43:57,916][INFO][o.e.p.PluginsService] [elastic] loaded module [frozen-indices] [2022-09-13T16:43:57,917][INFO][o.e.p.PluginsService] [elastic] loaded module [ingest-common] [2022-09-13T16:43:57,917][INFO][o.e.p.PluginsService] [elastic] loaded module [ingest-geoip] [2022-09-13T16:43:57,917][INFO][o.e.p.PluginsService] [elastic] loaded module [ingest-user-agent] [2022-09-13T16:43:57,917][INFO][o.e.p.PluginsService] [elastic] loaded module [kibana] [2022-09-13T16:43:57,918][INFO][o.e.p.PluginsService] [elastic] loaded module [lang-expression] [2022-09-13T16:43:57,918][INFO][o.e.p.PluginsService] [elastic] loaded module [lang-mustache] [2022-09-13T16:43:57,918][INFO][o.e.p.PluginsService] [elastic] loaded module [lang-painless] [2022-09-13T16:43:57,919][INFO][o.e.p.PluginsService] [elastic] loaded module [mapper-extras] [2022-09-13T16:43:57,919][INFO][o.e.p.PluginsService] [elastic] loaded module [mapper-version] [2022-09-13T16:43:57,919][INFO][o.e.p.PluginsService] [elastic] loaded module [parent-join] [2022-09-13T16:43:57,919][INFO][o.e.p.PluginsService] [elastic] loaded module [percolator] [2022-09-13T16:43:57,920][INFO][o.e.p.PluginsService] [elastic] loaded module [rank-eval] [2022-09-13T16:43:57,920][INFO][o.e.p.PluginsService] [elastic] loaded module [reindex] [2022-09-13T16:43:57,920][INFO][o.e.p.PluginsService] [elastic] loaded module [repositories-metering-api] [2022-09-13T16:43:57,921][INFO][o.e.p.PluginsService] [elastic] loaded module [repository-url] [2022-09-13T16:43:57,921][INFO][o.e.p.PluginsService] [elastic] loaded module [search-business-rules] [2022-09-13T16:43:57,921][INFO][o.e.p.PluginsService] [elastic] loaded module [searchable-snapshots] [2022-09-13T16:43:57,921][INFO][o.e.p.PluginsService] [elastic] loaded module [spatial] [2022-09-13T16:43:57,922][INFO][o.e.p.PluginsService] [elastic] loaded module [systemd] [2022-09-13T16:43:57,922][INFO][o.e.p.PluginsService] [elastic] loaded module [transform] [2022-09-13T16:43:57,922][INFO][o.e.p.PluginsService] [elastic] loaded module [transport-netty4] [2022-09-13T16:43:57,922][INFO][o.e.p.PluginsService] [elastic] loaded module [unsigned-long] [2022-09-13T16:43:57,923][INFO][o.e.p.PluginsService] [elastic] loaded module [vectors] [2022-09-13T16:43:57,923][INFO][o.e.p.PluginsService] [elastic] loaded module [wildcard] [2022-09-13T16:43:57,923][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-analytics] [2022-09-13T16:43:57,924][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-async] [2022-09-13T16:43:57,924][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-async-search] [2022-09-13T16:43:57,924][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-autoscaling] [2022-09-13T16:43:57,925][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-ccr] [2022-09-13T16:43:57,925][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-core] [2022-09-13T16:43:57,925][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-data-streams] [2022-09-13T16:43:57,925][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-deprecation] [2022-09-13T16:43:57,926][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-enrich] [2022-09-13T16:43:57,926][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-eql] [2022-09-13T16:43:57,926][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-graph] [2022-09-13T16:43:57,927][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-identity-provider] [2022-09-13T16:43:57,927][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-ilm] [2022-09-13T16:43:57,927][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-logstash] [2022-09-13T16:43:57,927][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-ml] [2022-09-13T16:43:57,928][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-monitoring] [2022-09-13T16:43:57,928][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-ql] [2022-09-13T16:43:57,928][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-rollup] [2022-09-13T16:43:57,928][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-security] [2022-09-13T16:43:57,929][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-sql] [2022-09-13T16:43:57,929][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-stack] [2022-09-13T16:43:57,929][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-voting-only-node] [2022-09-13T16:43:57,929][INFO][o.e.p.PluginsService] [elastic] loaded module [x-pack-watcher] [2022-09-13T16:43:57,930][INFO][o.e.p.PluginsService] [elastic] no plugins loaded [2022-09-13T16:43:57,969][INFO][o.e.e.NodeEnvironment] [elastic] using [1] data paths, mounts [[/graylog-data (/dev/mapper/graylog--data--vg-graylog--data)]], net usable_space [409.4gb], net total_space [806gb], types [ext4] [2022-09-13T16:43:57,969][INFO][o.e.e.NodeEnvironment] [elastic] heap size [8gb], compressed ordinary object pointers [true] [2022-09-13T16:43:58,131][INFO][o.e.n.Node] [elastic] node name [elastic], node ID [jXplRySTQxW_WDlSA02XUw], cluster name [graylog], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest] [2022-09-13T16:44:01,834][INFO][o.e.x.m.p.l.CppLogMessageHandler] [elastic] [controller/24093] [Main.cc@114] controller (64 bit): Version 7.10.2 (Build 40a3af639d4698) Copyright (c) 2021 Elasticsearch BV [2022-09-13T16:44:02,776][INFO][o.e.t.NettyAllocator] [elastic] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}] [2022-09-13T16:44:02,854][INFO][o.e.d.DiscoveryModule] [elastic] using discovery type [single-node] and seed hosts providers [settings] [2022-09-13T16:44:03,260][WARN][o.e.g.DanglingIndicesState] [elastic] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2022-09-13T16:44:03,605][INFO][o.e.n.Node] [elastic] initialized [2022-09-13T16:44:03,607][INFO][o.e.n.Node] [elastic] starting ... [2022-09-13T16:44:03,726][INFO][o.e.t.TransportService] [elastic] publish_address {10.0.221.16:9300}, bound_addresses {0.0.0.0:9300} [2022-09-13T16:44:03,978][INFO][o.e.c.c.Coordinator] [elastic] cluster UUID [04NbFYPNQXm9OtdD5yFE2w] [2022-09-13T16:44:04,121][INFO][o.e.c.s.MasterService] [elastic] elected-as-master ([1] nodes joined)[{elastic}{jXplRySTQxW_WDlSA02XUw}{FD1twWNDSSmoPh3Ahogmew}{10.0.221.16}{10.0.221.16:9300}{cdhilmrstw}{ml.machine_memory=8334688256, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 2, version: 117, delta: master node changed {previous [], current [{elastic}{jXplRySTQxW_WDlSA02XUw}{FD1twWNDSSmoPh3Ahogmew}{10.0.221.16}{10.0.221.16:9300}{cdhilmrstw}{ml.machine_memory=8334688256, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]} [2022-09-13T16:44:04,185][INFO][o.e.c.s.ClusterApplierService] [elastic] master node changed {previous [], current [{elastic}{jXplRySTQxW_WDlSA02XUw}{FD1twWNDSSmoPh3Ahogmew}{10.0.221.16}{10.0.221.16:9300}{cdhilmrstw}{ml.machine_memory=8334688256, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 2, version: 117, reason: Publication{term=2, version=117} [2022-09-13T16:44:04,221][INFO][o.e.h.AbstractHttpServerTransport] [elastic] publish_address {10.0.221.16:9200}, bound_addresses {0.0.0.0:9200} [2022-09-13T16:44:04,221][INFO][o.e.n.Node] [elastic] started [2022-09-13T16:44:04,542][INFO][o.e.l.LicenseService] [elastic] license [4e52af05-bf04-4329-aaed-6ac3d1824ef9] mode [basic] - valid [2022-09-13T16:44:04,546][INFO][o.e.g.GatewayService] [elastic] recovered [5] indices into cluster_state [2022-09-13T16:44:05,643][INFO][o.e.c.r.a.AllocationService] [elastic] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]]]). ``` elastic health ```auto curl -X GET "localhost:9200/_cluster/health?pretty" { "cluster_name" : "graylog", "status" : "green", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 18, "active_shards" : 18, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } ``` Does anyone have an idea what might be wrong or where I might check why the processing isn’t working? ---
### Author: ![ihe](https://avatars.discourse-cdn.com/v4/letter/i/a88e57/32.png) [@ihe](https://community.graylog.org/u/ihe) #### Post date: [September 13, 2022, 2:57pm UTC](https://community.graylog.org/t/graylog-not-processing-to-elasticseach-anymore/25700/4 "2022-09-13T14:57:05Z")
Is there a hell of CPU load on that system and no/very slow processing? Then I would assume a runaway grok. Java-Heap seams to be fine for Graylog, and as the output buffer is not filled I would not point in the direction of Elastic. Did you upgrade both, Graylog and ubuntu, at the same time? Maybe a processbuffer-dump or a thread-dump can explain what is going on. You will find it in the node overview. ![grafik](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/a/a93c6e4c5062ae71d4da56132bcfbc5e2f2bf0e4.png) ---
### Author: ![chas0rde](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/chas0rde/32/9035_2.png) [@chas0rde](https://community.graylog.org/u/chas0rde) #### Post date: [September 14, 2022, 6:19am UTC](https://community.graylog.org/t/graylog-not-processing-to-elasticseach-anymore/25700/5 "2022-09-14T06:19:01Z")
Hello high load is the java process of graylog utilizing 825% ```auto 28576 graylog 20 0 20.0g 6.9g 21272 S 825.6 59.2 7472:03 java ``` How would I find a runaway grok? Processbuffer dump is huge. What would I look for in it? Thread dump (shortened) Part I ```auto "kafka-journal-scheduler-1" id=331 state=TIMED_WAITING - waiting on <0x76ac2e25> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x76ac2e25> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-31" id=286 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-30" id=285 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-29" id=284 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-28" id=283 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "metrics-meter-tick-thread-2" id=282 state=WAITING - waiting on <0x2622fe96> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x2622fe96> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-27" id=281 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-26" id=279 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-25" id=277 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "metrics-meter-tick-thread-1" id=276 state=TIMED_WAITING - waiting on <0x2622fe96> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x2622fe96> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "kafka-journal-scheduler-0" id=275 state=WAITING - waiting on <0x76ac2e25> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x76ac2e25> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-24" id=271 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-23" id=267 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-22" id=266 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-21" id=265 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-20" id=264 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-19" id=263 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-18" id=262 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-17" id=261 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-16" id=259 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-15" id=258 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-14" id=257 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-13" id=256 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-12" id=255 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-11" id=254 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-10" id=252 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-9" id=250 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-8" id=248 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-7" id=247 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-6" id=246 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-5" id=245 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-29" id=243 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-4" id=242 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-28" id=241 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-27" id=240 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-3" id=239 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-26" id=238 state=TIMED_WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-25" id=237 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-24" id=236 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-23" id=235 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "http-worker-15" id=234 state=WAITING - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "http-worker-14" id=233 state=WAITING - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "http-worker-13" id=232 state=WAITING - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "http-worker-12" id=231 state=WAITING - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "http-worker-11" id=230 state=WAITING - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "http-worker-10" id=229 state=RUNNABLE (running in native) Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@3d34291c "proxied-requests-pool-2" id=228 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "http-worker-9" id=227 state=WAITING - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-22" id=225 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-21" id=224 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "netty-transport-1" id=223 state=RUNNABLE Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@21f030ed "http-worker-8" id=222 state=WAITING - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "http-worker-7" id=221 state=WAITING - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "http-worker-6" id=220 state=WAITING - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "http-worker-5" id=219 state=WAITING - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "http-worker-4" id=218 state=RUNNABLE Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@4d32358a "proxied-requests-pool-1" id=217 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "proxied-requests-pool-0" id=216 state=WAITING - waiting on <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7aee608e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "http-worker-3" id=215 state=WAITING - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-20" id=214 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-19" id=213 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "SessionValidationThread-1" id=212 state=TIMED_WAITING - waiting on <0x42d58ad4> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x42d58ad4> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "http-worker-2" id=211 state=WAITING - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "http-worker-1" id=210 state=WAITING - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-18" id=209 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "http-worker-0" id=208 state=WAITING - waiting on <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x3ddda076> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-17" id=207 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbuffer-processor-executor-2" id=204 state=WAITING - waiting on <0x26f54779> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x26f54779> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbuffer-processor-executor-2" id=203 state=WAITING - waiting on <0x51e5d253> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x51e5d253> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbuffer-processor-executor-2" id=202 state=WAITING - waiting on <0x7f48f971> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7f48f971> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbuffer-processor-executor-2" id=201 state=WAITING - waiting on <0x21775881> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x21775881> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbuffer-processor-executor-2" id=200 state=WAITING - waiting on <0x364de524> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x364de524> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbuffer-processor-executor-1" id=199 state=WAITING - waiting on <0x26f54779> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x26f54779> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbuffer-processor-executor-1" id=197 state=WAITING - waiting on <0x51e5d253> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x51e5d253> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbuffer-processor-executor-1" id=196 state=WAITING - waiting on <0x7f48f971> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7f48f971> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbuffer-processor-executor-1" id=195 state=WAITING - waiting on <0x21775881> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x21775881> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbuffer-processor-executor-1" id=194 state=WAITING - waiting on <0x364de524> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x364de524> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbuffer-processor-executor-0" id=193 state=WAITING - waiting on <0x26f54779> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x26f54779> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbuffer-processor-executor-0" id=190 state=WAITING - waiting on <0x7f48f971> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7f48f971> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbuffer-processor-executor-0" id=191 state=WAITING - waiting on <0x51e5d253> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x51e5d253> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "netty-transport-0" id=187 state=RUNNABLE Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@3b441bd0 "outputbuffer-processor-executor-0" id=185 state=WAITING - waiting on <0x21775881> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x21775881> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbuffer-processor-executor-0" id=186 state=WAITING - waiting on <0x364de524> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x364de524> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-16" id=182 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-15" id=181 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "netty-transport-1" id=180 state=RUNNABLE Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@2f2cd210 "netty-transport-3" id=179 state=RUNNABLE (running in native) Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@6ed6bf86 "netty-transport-0" id=178 state=RUNNABLE Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@7f62be6b "netty-transport-2" id=175 state=RUNNABLE Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@aa9f913 "netty-transport-1" id=174 state=RUNNABLE Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@3357337a "netty-transport-0" id=173 state=RUNNABLE Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@6f598717 "grizzly-nio-kernel(1) SelectorRunner" id=172 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@3812c23c "HttpServer-0" id=171 state=TIMED_WAITING - waiting on <0x651e566a> (a java.lang.Object) - locked <0x651e566a> (a java.lang.Object) Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@9222702 "scheduled-14" id=170 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-13" id=169 state=TIMED_WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-12" id=168 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-11" id=167 state=TIMED_WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "nioEventLoopGroup-2-1" id=166 state=RUNNABLE (running in native) - locked sun.nio.ch.EPollSelectorImpl@7b9ea9eb "scheduled-10" id=165 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-9" id=154 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-8" id=153 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-29" id=151 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-28" id=150 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-27" id=149 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-26" id=148 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-25" id=147 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-7" id=146 state=TIMED_WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-24" id=145 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-23" id=144 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-22" id=142 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-21" id=141 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-20" id=140 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-19" id=139 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-18" id=138 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-17" id=137 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-16" id=136 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-15" id=135 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "job-heartbeat-0" id=134 state=TIMED_WAITING - waiting on <0x6798a455> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x6798a455> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-14" id=133 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-13" id=131 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-12" id=130 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-11" id=129 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-10" id=128 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-9" id=127 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-8" id=126 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-6" id=125 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-7" id=124 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-6" id=123 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-5" id=122 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-5" id=119 state=TIMED_WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-4" id=116 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "JobSchedulerService" id=113 state=TIMED_WAITING - waiting on <0x4aea8e9c> (a java.util.concurrent.Semaphore$NonfairSync) - locked <0x4aea8e9c> (a java.util.concurrent.Semaphore$NonfairSync) ``` Seems most stuff is waiting (though it seems according to the CPU load maybe busy-waiting) Release upgrade was done on both systems on the same day ---
### Author: ![chas0rde](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/chas0rde/32/9035_2.png) [@chas0rde](https://community.graylog.org/u/chas0rde) #### Post date: [September 14, 2022, 6:19am UTC](https://community.graylog.org/t/graylog-not-processing-to-elasticseach-anymore/25700/6 "2022-09-14T06:19:17Z")
Part II ```auto "scheduled-4" id=112 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "output-setup-service-0" id=108 state=WAITING - waiting on <0x523d8b41> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x523d8b41> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-3" id=107 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-2" id=105 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-1" id=103 state=TIMED_WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-3" id=102 state=TIMED_WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-0" id=100 state=WAITING - waiting on <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x473cb837> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "LocalKafkaMessageQueueReader" id=95 state=TIMED_WAITING "InputSetupService" id=90 state=WAITING - waiting on <0x483cbc68> (a java.util.concurrent.CountDownLatch$Sync) - locked <0x483cbc68> (a java.util.concurrent.CountDownLatch$Sync) "FailureHandlingService" id=88 state=WAITING - waiting on <0x504900ca> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x504900ca> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "cluster-eventbus-handler-1" id=85 state=WAITING - waiting on <0x619dd25a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x619dd25a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "eventbus-handler-1" id=84 state=WAITING - waiting on <0x7f6d5f3e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7f6d5f3e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "eventbus-handler-0" id=83 state=WAITING - waiting on <0x7f6d5f3e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7f6d5f3e> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "cluster-eventbus-handler-0" id=82 state=WAITING - waiting on <0x619dd25a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x619dd25a> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-2" id=81 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-1" id=80 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "processbufferprocessor-7" id=79 state=RUNNABLE "processbufferprocessor-6" id=78 state=RUNNABLE "processbufferprocessor-5" id=77 state=RUNNABLE "processbufferprocessor-4" id=76 state=RUNNABLE "processbufferprocessor-3" id=75 state=RUNNABLE "processbufferprocessor-2" id=74 state=RUNNABLE "processbufferprocessor-1" id=73 state=RUNNABLE "processbufferprocessor-0" id=72 state=RUNNABLE "aws-instance-lookup-refresher-0" id=71 state=TIMED_WAITING - waiting on <0x0ab24be0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x0ab24be0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "aws-instance-lookup-refresher-0" id=70 state=TIMED_WAITING - waiting on <0x21d1db52> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x21d1db52> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "aws-instance-lookup-refresher-0" id=69 state=TIMED_WAITING - waiting on <0x27173f28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x27173f28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "aws-instance-lookup-refresher-0" id=68 state=TIMED_WAITING - waiting on <0x6521dcb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x6521dcb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "aws-instance-lookup-refresher-0" id=67 state=TIMED_WAITING - waiting on <0x58ae6cc4> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x58ae6cc4> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "aws-instance-lookup-refresher-0" id=66 state=TIMED_WAITING - waiting on <0x2d4570a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x2d4570a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "aws-instance-lookup-refresher-0" id=65 state=TIMED_WAITING - waiting on <0x0be933b7> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x0be933b7> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbufferprocessor-4" id=64 state=WAITING - waiting on <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbufferprocessor-3" id=63 state=WAITING - waiting on <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbufferprocessor-2" id=62 state=WAITING - waiting on <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbufferprocessor-1" id=61 state=WAITING - waiting on <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "outputbufferprocessor-0" id=60 state=WAITING - waiting on <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x7379d978> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "aws-instance-lookup-refresher-0" id=59 state=TIMED_WAITING - waiting on <0x75e01fc3> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x75e01fc3> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "scheduled-daemon-0" id=58 state=WAITING - waiting on <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x5a4f98aa> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "I/O dispatcher 20" id=57 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@33897c3c "I/O dispatcher 19" id=56 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@65ecac "I/O dispatcher 18" id=55 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@5a98a565 "I/O dispatcher 17" id=54 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@2fb703e "I/O dispatcher 16" id=53 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@77854aaa "I/O dispatcher 15" id=52 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@1d39f74e "I/O dispatcher 14" id=51 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@545e368f "I/O dispatcher 13" id=50 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@5771c1a6 "I/O dispatcher 12" id=49 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@5456fe8d "I/O dispatcher 11" id=48 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@2b9f916b "I/O dispatcher 10" id=47 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@7fcc505b "I/O dispatcher 9" id=46 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@66b50a4a "I/O dispatcher 8" id=45 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@6cafb46e "I/O dispatcher 7" id=44 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@a95537a "I/O dispatcher 6" id=43 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@2ecff4f0 "I/O dispatcher 5" id=42 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@6c822e24 "I/O dispatcher 4" id=41 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@1eadf5c3 "I/O dispatcher 3" id=40 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@2b08316e "I/O dispatcher 2" id=39 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@4ef144ab "I/O dispatcher 1" id=38 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@4d81a6eb "pool-9-thread-1" id=37 state=RUNNABLE - locked sun.nio.ch.EPollSelectorImpl@c95e691 "OkHttp ConnectionPool" id=36 state=TIMED_WAITING - waiting on <0x7412985f> (a okhttp3.internal.connection.RealConnectionPool) - locked <0x7412985f> (a okhttp3.internal.connection.RealConnectionPool) Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@269a37bc "inputbufferprocessor-2" id=35 state=WAITING - waiting on <0x6a897190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x6a897190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "inputbufferprocessor-1" id=34 state=WAITING - waiting on <0x6a897190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x6a897190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "inputbufferprocessor-0" id=33 state=WAITING - waiting on <0x6a897190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x6a897190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "CleanCursors-2-thread-1" id=32 state=TIMED_WAITING - waiting on <0x3e3a695b> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x3e3a695b> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "cluster-ClusterId{value='6320976d2ff9f01e48bb4ff6', description='null'}-localhost:27017" id=31 state=TIMED_WAITING - waiting on <0x45da0aa5> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) - locked <0x45da0aa5> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) "Okio Watchdog" id=28 state=TIMED_WAITING - waiting on <0x10b4e7f8> (a java.lang.Class) - locked <0x10b4e7f8> (a java.lang.Class) "Common-Cleaner" id=19 state=TIMED_WAITING - waiting on <0x562058f4> (a java.lang.ref.ReferenceQueue$Lock) - locked <0x562058f4> (a java.lang.ref.ReferenceQueue$Lock) "Signal Dispatcher" id=4 state=RUNNABLE "Finalizer" id=3 state=WAITING - waiting on <0x095e58b5> (a java.lang.ref.ReferenceQueue$Lock) - locked <0x095e58b5> (a java.lang.ref.ReferenceQueue$Lock) "Reference Handler" id=2 state=RUNNABLE "main" id=1 state=WAITING - waiting on <0x79d743e6> (a java.lang.Thread) - locked <0x79d743e6> (a java.lang.Thread) ``` ---
### Author: ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith) #### Post date: [September 19, 2022, 9:48pm UTC](https://community.graylog.org/t/graylog-not-processing-to-elasticseach-anymore/25700/7 "2022-09-19T21:48:07Z")
Hello, Do you have any pipelines or extractors on this Graylog Server? If so, for troubleshooting I would looking into disabling or remove them to see if it clears up. ---
### Author: ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system) #### Post date: [October 3, 2022, 9:48pm UTC](https://community.graylog.org/t/graylog-not-processing-to-elasticseach-anymore/25700/8 "2022-10-03T21:48:09Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.