Graylog doesn't process messages anymore, ELS cluster is green

Dear community,

first of all, thank you for the ability to ask you guys.
This is my current setup

  • 3x Elasticsearch Nodes VMs via docker-compose
    (Each: 32 RAM / 16 CPU Cores / 42 TB HDD)
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2

  • 2x Graylog Nodes VMs via docker-compose
    (Each: 16 RAM / 8 CPU / 30 GB HDD)
    image: graylog/graylog:4.0.3

  • 3x MongoDB Instances (on each graylog + on 1 elasticsearch node)
    image: mongo:latest (4.2.12)

Everything worked fine until 22th this month.
Graylog just stopped processing messages.

See the screenshot taken here:

I already read a lot of threads here with topic “unprocessed messages”.
Most of the root causes in that threads were performance issues or no disk space left.
But I dont think I have performance issues.

Here is the graylog Indices tab:

Here is the graylog Overview tab:


As you can see my Elasticsearch Cluster is green and healthy.

I checked Elasticsearch resources but everything is low. Nothing to process.

I already tried to shutdown all graylog nodes and delete graylog_journal but without success:
Graylog is still not processing any messages.

Any help is very appreciated!
If you need some more infos please let me now.

Best regards
Hollowdew

Check /var/log/elasticsearch/graylog.log for any errors.

Dear Ponet Jesse Hills,

thank you for your reply.
All Services are running in docker.
I got a bash from the elasticsearch container but there is no directory /var/log/elasticsearch/…
All logs should be stdout ,stderr.

So I did docker logs -f elasticsearch

{“type”: “server”, “timestamp”: “2021-03-25T08:49:55,791Z”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “master node changed {previous , current [{elastic-node3.my.domain.com}{6QYuggRKRkueYuH_pOfb3g}{tm0bHrK1T–rLHSXdVgM9Q}{192.168.10.216}{192.168.10.216:9300}{dimr}]}, term: 137, version: 469, reason: ApplyCommitRequest{term=137, version=469, sourceNode={elastic-node3.my.domain.com}{6QYuggRKRkueYuH_pOfb3g}{tm0bHrK1T–rLHSXdVgM9Q}{192.168.10.216}{192.168.10.216:9300}{dimr}}”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T08:49:55,854Z”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “removed {{elastic-node2.my.domain.com}{RgOq1qp8Rxa4i1FFpHdbqA}{RcRy5QYTTqWkIHb4sFTrXg}{192.168.10.215}{192.168.10.215:9300}{dimr}}, term: 137, version: 470, reason: ApplyCommitRequest{term=137, version=470, sourceNode={elastic-node3.my.domain.com}{6QYuggRKRkueYuH_pOfb3g}{tm0bHrK1T–rLHSXdVgM9Q}{192.168.10.216}{192.168.10.216:9300}{dimr}}”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T08:50:03,568Z”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “added {{elastic-node2.my.domain.com}{RgOq1qp8Rxa4i1FFpHdbqA}{Tfo8VZkVTLK2k8m3OAwqhA}{192.168.10.215}{192.168.10.215:9300}{dimr}}, term: 137, version: 472, reason: ApplyCommitRequest{term=137, version=472, sourceNode={elastic-node3.my.domain.com}{6QYuggRKRkueYuH_pOfb3g}{tm0bHrK1T–rLHSXdVgM9Q}{192.168.10.216}{192.168.10.216:9300}{dimr}}”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:10,831Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “stopping …”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:10,837Z”, “level”: “INFO”, “component”: “o.e.c.c.Coordinator”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “master node [{elastic-node3.my.domain.com}{6QYuggRKRkueYuH_pOfb3g}{tm0bHrK1T–rLHSXdVgM9Q}{192.168.10.216}{192.168.10.216:9300}{dimr}] failed, restarting discovery”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” ,
“stacktrace”: [“org.elasticsearch.transport.NodeDisconnectedException: [elastic-node3.my.domain.com][192.168.10.216:9300][disconnected] disconnected”] }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:10,838Z”, “level”: “DEBUG”, “component”: “o.e.d.SeedHostsResolver”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “resolveConfiguredHosts: lifecycle is STOPPED, not proceeding”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:10,844Z”, “level”: “DEBUG”, “component”: “o.e.d.PeerFinder”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “Peer{transportAddress=192.168.10.216:9300, discoveryNode=null, peersRequestInFlight=false} connection failed”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” ,
“stacktrace”: [“java.lang.IllegalStateException: transport has been stopped”,
“at org.elasticsearch.transport.TcpTransport.ensureOpen(TcpTransport.java:845) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:290) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.ClusterConnectionManager.internalOpenConnection(ClusterConnectionManager.java:254) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.ClusterConnectionManager.openConnection(ClusterConnectionManager.java:95) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:420) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1.doRun(HandshakingTransportAddressConnector.java:86) [elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) [elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]”,
“at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]”,
“at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]”,
“at java.lang.Thread.run(Thread.java:832) [?:?]”] }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:10,844Z”, “level”: “DEBUG”, “component”: “o.e.d.PeerFinder”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “Peer{transportAddress=192.168.10.215:9300, discoveryNode=null, peersRequestInFlight=false} connection failed”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” ,
“stacktrace”: [“java.lang.IllegalStateException: transport has been stopped”,
“at org.elasticsearch.transport.TcpTransport.ensureOpen(TcpTransport.java:845) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:290) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.ClusterConnectionManager.internalOpenConnection(ClusterConnectionManager.java:254) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.ClusterConnectionManager.openConnection(ClusterConnectionManager.java:95) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:420) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1.doRun(HandshakingTransportAddressConnector.java:86) [elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) [elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]”,
“at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]”,
“at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]”,
“at java.lang.Thread.run(Thread.java:832) [?:?]”] }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:11,020Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “stopped”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:11,021Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “closing …”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:11,030Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “closed”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
[INFO tini (1)] Spawned child process ‘/usr/local/bin/docker-entrypoint.sh’ with pid ‘7’
[INFO tini (1)] Main child exited normally (with status ‘143’)
{“type”: “server”, “timestamp”: “2021-03-25T09:04:17,674Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “version[7.10.2], pid[7], build[oss/docker/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.19.0-13-amd64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:17,676Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:17,676Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-8395086767268250659, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms20480m, -Xmx20480m, -XX:MaxDirectMemorySize=10737418240, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=docker, -Des.bundled_jdk=true]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,234Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [aggs-matrix-stats]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,234Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [analysis-common]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,235Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [geo]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,235Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [ingest-common]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,235Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [ingest-geoip]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,235Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [ingest-user-agent]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,235Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [kibana]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,236Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [lang-expression]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,236Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [lang-mustache]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,236Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [lang-painless]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,236Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [mapper-extras]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,236Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [parent-join]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,237Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [percolator]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,237Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [rank-eval]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,237Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [reindex]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,237Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [repository-url]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,237Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [transport-netty4]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,238Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “no plugins loaded” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,274Z”, “level”: “INFO”, “component”: “o.e.e.NodeEnvironment”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “using [1] data paths, mounts [[/usr/share/elasticsearch/data (:/volume1/elasticsearch/node1)]], net usable_space [46.8tb], net total_space [46.8tb], types [nfs]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,275Z”, “level”: “INFO”, “component”: “o.e.e.NodeEnvironment”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “heap size [20gb], compressed ordinary object pointers [true]” }
{“type”: “deprecation”, “timestamp”: “2021-03-25T09:04:18,298Z”, “level”: “DEPRECATION”, “component”: “o.e.d.c.s.Settings”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “[node.master] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.” }
{“type”: “deprecation”, “timestamp”: “2021-03-25T09:04:18,304Z”, “level”: “DEPRECATION”, “component”: “o.e.d.c.s.Settings”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “[node.data] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,377Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “node name [elastic-node1.my.domain.com], node ID [TptkW61sT96vQOAWvI4Qmg], cluster name [graylog], roles [master, remote_cluster_client, data, ingest]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:19,632Z”, “level”: “DEBUG”, “component”: “o.e.d.z.ElectMasterService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “using minimum_master_nodes [-1]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:20,541Z”, “level”: “INFO”, “component”: “o.e.t.NettyAllocator”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:20,574Z”, “level”: “DEBUG”, “component”: “o.e.d.SettingsBasedSeedHostsProvider”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “using initial hosts [elastic-node2.my.domain.com, elastic-node3.my.domain.com]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:20,595Z”, “level”: “INFO”, “component”: “o.e.d.DiscoveryModule”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “using discovery type [zen] and seed hosts providers [settings]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:20,793Z”, “level”: “WARN”, “component”: “o.e.g.DanglingIndicesState”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:20,910Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “initialized” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:20,910Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “starting …” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:20,984Z”, “level”: “INFO”, “component”: “o.e.t.TransportService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “publish_address {192.168.10.214:9300}, bound_addresses {0.0.0.0:9300}” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:21,143Z”, “level”: “INFO”, “component”: “o.e.b.BootstrapChecks”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “bound or publishing to a non-loopback address, enforcing bootstrap checks” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:21,144Z”, “level”: “DEBUG”, “component”: “o.e.d.SeedHostsResolver”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “using max_concurrent_resolvers [10], resolver timeout [5s]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:21,145Z”, “level”: “INFO”, “component”: “o.e.c.c.Coordinator”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “cluster UUID [_WWVJSF9S4KCVXcTUR7Smg]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:21,357Z”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “master node changed {previous , current [{elastic-node2.my.domain.com}{RgOq1qp8Rxa4i1FFpHdbqA}{-WKkKPbaQJ-qjOdwqQ_tVw}{192.168.10.215}{192.168.10.215:9300}{dimr}]}, added {{elastic-node2.my.domain.com}{RgOq1qp8Rxa4i1FFpHdbqA}{-WKkKPbaQJ-qjOdwqQ_tVw}{192.168.10.215}{192.168.10.215:9300}{dimr},{elastic-node3.my.domain.com}{6QYuggRKRkueYuH_pOfb3g}{2mVQSIFSTHiIHbskfLhbzA}{192.168.10.216}{192.168.10.216:9300}{dimr}}, term: 138, version: 483, reason: ApplyCommitRequest{term=138, version=483, sourceNode={elastic-node2.my.domain.com}{RgOq1qp8Rxa4i1FFpHdbqA}{-WKkKPbaQJ-qjOdwqQ_tVw}{192.168.10.215}{192.168.10.215:9300}{dimr}}” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:21,382Z”, “level”: “INFO”, “component”: “o.e.h.AbstractHttpServerTransport”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “publish_address {192.168.16.3:9200}, bound_addresses {0.0.0.0:9200}”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:21,383Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “started”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }

I already tried:

  • shutdown graylog nodes → delete graylog_journal
  • restart all graylog nodes and elasticsearch nodes
  • graceful shutdown both graylog nodes via WebUI
  • stop processing → start processing on both nodes
  • stop all inputs → start all inputs

Finally graylog processing messages.
I just exported and deleted all extractors from all inputs.
I only use GROK Extractors.
Where can I debug that?
On of the Extractors put all graylog nodes to freeze and no processing messages.

might’ve been something similar to this thread and this idea.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.