Graylog doesn't process messages anymore, ELS cluster is green

Graylog Central (peer support)
1

Dear community,

first of all, thank you for the ability to ask you guys.
This is my current setup

  • 3x Elasticsearch Nodes VMs via docker-compose
    (Each: 32 RAM / 16 CPU Cores / 42 TB HDD)
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2

  • 2x Graylog Nodes VMs via docker-compose
    (Each: 16 RAM / 8 CPU / 30 GB HDD)
    image: graylog/graylog:4.0.3

  • 3x MongoDB Instances (on each graylog + on 1 elasticsearch node)
    image: mongo:latest (4.2.12)

Everything worked fine until 22th this month.
Graylog just stopped processing messages.

See the screenshot taken here:

image
image1889×360 47.4 KB

I already read a lot of threads here with topic “unprocessed messages”.
Most of the root causes in that threads were performance issues or no disk space left.
But I dont think I have performance issues.

Here is the graylog Indices tab:

image
image1903×824 75.8 KB

Here is the graylog Overview tab:

image
image1914×1055 81.8 KB

As you can see my Elasticsearch Cluster is green and healthy.

I checked Elasticsearch resources but everything is low. Nothing to process.

I already tried to shutdown all graylog nodes and delete graylog_journal but without success:
Graylog is still not processing any messages.

Any help is very appreciated!
If you need some more infos please let me now.

Best regards
Hollowdew

2

Check /var/log/elasticsearch/graylog.log for any errors.

3

Dear Ponet Jesse Hills,

thank you for your reply.
All Services are running in docker.
I got a bash from the elasticsearch container but there is no directory /var/log/elasticsearch/…
All logs should be stdout ,stderr.

So I did docker logs -f elasticsearch

{“type”: “server”, “timestamp”: “2021-03-25T08:49:55,791Z”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “master node changed {previous , current [{elastic-node3.my.domain.com}{6QYuggRKRkueYuH_pOfb3g}{tm0bHrK1T–rLHSXdVgM9Q}{192.168.10.216}{192.168.10.216:9300}{dimr}]}, term: 137, version: 469, reason: ApplyCommitRequest{term=137, version=469, sourceNode={elastic-node3.my.domain.com}{6QYuggRKRkueYuH_pOfb3g}{tm0bHrK1T–rLHSXdVgM9Q}{192.168.10.216}{192.168.10.216:9300}{dimr}}”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T08:49:55,854Z”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “removed {{elastic-node2.my.domain.com}{RgOq1qp8Rxa4i1FFpHdbqA}{RcRy5QYTTqWkIHb4sFTrXg}{192.168.10.215}{192.168.10.215:9300}{dimr}}, term: 137, version: 470, reason: ApplyCommitRequest{term=137, version=470, sourceNode={elastic-node3.my.domain.com}{6QYuggRKRkueYuH_pOfb3g}{tm0bHrK1T–rLHSXdVgM9Q}{192.168.10.216}{192.168.10.216:9300}{dimr}}”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T08:50:03,568Z”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “added {{elastic-node2.my.domain.com}{RgOq1qp8Rxa4i1FFpHdbqA}{Tfo8VZkVTLK2k8m3OAwqhA}{192.168.10.215}{192.168.10.215:9300}{dimr}}, term: 137, version: 472, reason: ApplyCommitRequest{term=137, version=472, sourceNode={elastic-node3.my.domain.com}{6QYuggRKRkueYuH_pOfb3g}{tm0bHrK1T–rLHSXdVgM9Q}{192.168.10.216}{192.168.10.216:9300}{dimr}}”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:10,831Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “stopping …”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:10,837Z”, “level”: “INFO”, “component”: “o.e.c.c.Coordinator”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “master node [{elastic-node3.my.domain.com}{6QYuggRKRkueYuH_pOfb3g}{tm0bHrK1T–rLHSXdVgM9Q}{192.168.10.216}{192.168.10.216:9300}{dimr}] failed, restarting discovery”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” ,
“stacktrace”: [“org.elasticsearch.transport.NodeDisconnectedException: [elastic-node3.my.domain.com][192.168.10.216:9300][disconnected] disconnected”] }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:10,838Z”, “level”: “DEBUG”, “component”: “o.e.d.SeedHostsResolver”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “resolveConfiguredHosts: lifecycle is STOPPED, not proceeding”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:10,844Z”, “level”: “DEBUG”, “component”: “o.e.d.PeerFinder”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “Peer{transportAddress=192.168.10.216:9300, discoveryNode=null, peersRequestInFlight=false} connection failed”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” ,
“stacktrace”: [“java.lang.IllegalStateException: transport has been stopped”,
“at org.elasticsearch.transport.TcpTransport.ensureOpen(TcpTransport.java:845) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:290) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.ClusterConnectionManager.internalOpenConnection(ClusterConnectionManager.java:254) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.ClusterConnectionManager.openConnection(ClusterConnectionManager.java:95) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:420) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1.doRun(HandshakingTransportAddressConnector.java:86) [elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) [elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]”,
“at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]”,
“at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]”,
“at java.lang.Thread.run(Thread.java:832) [?:?]”] }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:10,844Z”, “level”: “DEBUG”, “component”: “o.e.d.PeerFinder”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “Peer{transportAddress=192.168.10.215:9300, discoveryNode=null, peersRequestInFlight=false} connection failed”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” ,
“stacktrace”: [“java.lang.IllegalStateException: transport has been stopped”,
“at org.elasticsearch.transport.TcpTransport.ensureOpen(TcpTransport.java:845) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:290) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.ClusterConnectionManager.internalOpenConnection(ClusterConnectionManager.java:254) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.ClusterConnectionManager.openConnection(ClusterConnectionManager.java:95) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:420) ~[elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1.doRun(HandshakingTransportAddressConnector.java:86) [elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) [elasticsearch-7.10.2.jar:7.10.2]”,
“at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]”,
“at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]”,
“at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]”,
“at java.lang.Thread.run(Thread.java:832) [?:?]”] }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:11,020Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “stopped”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:11,021Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “closing …”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T09:03:11,030Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “closed”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
[INFO tini (1)] Spawned child process ‘/usr/local/bin/docker-entrypoint.sh’ with pid ‘7’
[INFO tini (1)] Main child exited normally (with status ‘143’)
{“type”: “server”, “timestamp”: “2021-03-25T09:04:17,674Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “version[7.10.2], pid[7], build[oss/docker/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.19.0-13-amd64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:17,676Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:17,676Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-8395086767268250659, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms20480m, -Xmx20480m, -XX:MaxDirectMemorySize=10737418240, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=docker, -Des.bundled_jdk=true]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,234Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [aggs-matrix-stats]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,234Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [analysis-common]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,235Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [geo]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,235Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [ingest-common]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,235Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [ingest-geoip]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,235Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [ingest-user-agent]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,235Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [kibana]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,236Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [lang-expression]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,236Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [lang-mustache]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,236Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [lang-painless]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,236Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [mapper-extras]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,236Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [parent-join]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,237Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [percolator]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,237Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [rank-eval]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,237Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [reindex]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,237Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [repository-url]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,237Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “loaded module [transport-netty4]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,238Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “no plugins loaded” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,274Z”, “level”: “INFO”, “component”: “o.e.e.NodeEnvironment”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “using [1] data paths, mounts [[/usr/share/elasticsearch/data (:/volume1/elasticsearch/node1)]], net usable_space [46.8tb], net total_space [46.8tb], types [nfs]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,275Z”, “level”: “INFO”, “component”: “o.e.e.NodeEnvironment”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “heap size [20gb], compressed ordinary object pointers [true]” }
{“type”: “deprecation”, “timestamp”: “2021-03-25T09:04:18,298Z”, “level”: “DEPRECATION”, “component”: “o.e.d.c.s.Settings”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “[node.master] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.” }
{“type”: “deprecation”, “timestamp”: “2021-03-25T09:04:18,304Z”, “level”: “DEPRECATION”, “component”: “o.e.d.c.s.Settings”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “[node.data] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:18,377Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “node name [elastic-node1.my.domain.com], node ID [TptkW61sT96vQOAWvI4Qmg], cluster name [graylog], roles [master, remote_cluster_client, data, ingest]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:19,632Z”, “level”: “DEBUG”, “component”: “o.e.d.z.ElectMasterService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “using minimum_master_nodes [-1]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:20,541Z”, “level”: “INFO”, “component”: “o.e.t.NettyAllocator”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:20,574Z”, “level”: “DEBUG”, “component”: “o.e.d.SettingsBasedSeedHostsProvider”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “using initial hosts [elastic-node2.my.domain.com, elastic-node3.my.domain.com]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:20,595Z”, “level”: “INFO”, “component”: “o.e.d.DiscoveryModule”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “using discovery type [zen] and seed hosts providers [settings]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:20,793Z”, “level”: “WARN”, “component”: “o.e.g.DanglingIndicesState”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:20,910Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “initialized” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:20,910Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “starting …” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:20,984Z”, “level”: “INFO”, “component”: “o.e.t.TransportService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “publish_address {192.168.10.214:9300}, bound_addresses {0.0.0.0:9300}” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:21,143Z”, “level”: “INFO”, “component”: “o.e.b.BootstrapChecks”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “bound or publishing to a non-loopback address, enforcing bootstrap checks” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:21,144Z”, “level”: “DEBUG”, “component”: “o.e.d.SeedHostsResolver”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “using max_concurrent_resolvers [10], resolver timeout [5s]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:21,145Z”, “level”: “INFO”, “component”: “o.e.c.c.Coordinator”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “cluster UUID [_WWVJSF9S4KCVXcTUR7Smg]” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:21,357Z”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “master node changed {previous , current [{elastic-node2.my.domain.com}{RgOq1qp8Rxa4i1FFpHdbqA}{-WKkKPbaQJ-qjOdwqQ_tVw}{192.168.10.215}{192.168.10.215:9300}{dimr}]}, added {{elastic-node2.my.domain.com}{RgOq1qp8Rxa4i1FFpHdbqA}{-WKkKPbaQJ-qjOdwqQ_tVw}{192.168.10.215}{192.168.10.215:9300}{dimr},{elastic-node3.my.domain.com}{6QYuggRKRkueYuH_pOfb3g}{2mVQSIFSTHiIHbskfLhbzA}{192.168.10.216}{192.168.10.216:9300}{dimr}}, term: 138, version: 483, reason: ApplyCommitRequest{term=138, version=483, sourceNode={elastic-node2.my.domain.com}{RgOq1qp8Rxa4i1FFpHdbqA}{-WKkKPbaQJ-qjOdwqQ_tVw}{192.168.10.215}{192.168.10.215:9300}{dimr}}” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:21,382Z”, “level”: “INFO”, “component”: “o.e.h.AbstractHttpServerTransport”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “publish_address {192.168.16.3:9200}, bound_addresses {0.0.0.0:9200}”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }
{“type”: “server”, “timestamp”: “2021-03-25T09:04:21,383Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “elastic-node1.my.domain.com”, “message”: “started”, “cluster.uuid”: “_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” }

4

I already tried:

  • shutdown graylog nodes → delete graylog_journal
  • restart all graylog nodes and elasticsearch nodes
  • graceful shutdown both graylog nodes via WebUI
  • stop processing → start processing on both nodes
  • stop all inputs → start all inputs
5

Finally graylog processing messages.
I just exported and deleted all extractors from all inputs.
I only use GROK Extractors.
Where can I debug that?
On of the Extractors put all graylog nodes to freeze and no processing messages.

6

might’ve been something similar to this thread and this idea.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.