# Graylog doesn't process messages anymore, ELS cluster is green **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [March 25, 2021, 10:06am UTC](https://community.graylog.org/t/graylog-doesnt-process-messages-anymore-els-cluster-is-green/19252 "2021-03-25T10:06:25Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![hollowdew](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@hollowdew](https://community.graylog.org/u/hollowdew)\ **Post date:** [March 25, 2021, 10:06am UTC](https://community.graylog.org/t/graylog-doesnt-process-messages-anymore-els-cluster-is-green/19252/1 "2021-03-25T10:06:25Z")
Dear community, first of all, thank you for the ability to ask you guys. This is my current setup - 3x Elasticsearch Nodes VMs via docker-compose (Each: 32 RAM / 16 CPU Cores / 42 TB HDD) image: [docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2](http://docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2) - 2x Graylog Nodes VMs via docker-compose (Each: 16 RAM / 8 CPU / 30 GB HDD) image: graylog/graylog:4.0.3 - 3x MongoDB Instances (on each graylog + on 1 elasticsearch node) image: mongo:latest (4.2.12) Everything worked fine until 22th this month. Graylog just stopped processing messages. See the screenshot taken here: ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/5/53755c019c9ec1011de67dc5c64c4a206a0cac00.png) I already read a lot of threads here with topic “unprocessed messages”. Most of the root causes in that threads were performance issues or no disk space left. But I dont think I have performance issues. Here is the graylog Indices tab: ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/b/b2d5e54b992e61a187db9a31fb18173c39c41811.png) Here is the graylog Overview tab: ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/a/a6e96cd0d8b3915ca3eff13f5a7f19b9d8e55af9.png) As you can see my Elasticsearch Cluster is green and healthy. I checked Elasticsearch resources but everything is low. Nothing to process. I already tried to shutdown all graylog nodes and delete graylog\_journal but without success: Graylog is still not processing any messages. Any help is very appreciated! If you need some more infos please let me now. Best regards Hollowdew ---
**Author:** ![Ponet](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/ponet/32/2131_2.png) [@Ponet](https://community.graylog.org/u/Ponet)\ **Post date:** [March 25, 2021, 11:18am UTC](https://community.graylog.org/t/graylog-doesnt-process-messages-anymore-els-cluster-is-green/19252/2 "2021-03-25T11:18:25Z")
Check `/var/log/elasticsearch/graylog.log` for any errors. ---
**Author:** ![hollowdew](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@hollowdew](https://community.graylog.org/u/hollowdew)\ **Post date:** [March 25, 2021, 11:25am UTC](https://community.graylog.org/t/graylog-doesnt-process-messages-anymore-els-cluster-is-green/19252/3 "2021-03-25T11:25:30Z")
Dear Ponet Jesse Hills, thank you for your reply. All Services are running in docker. I got a bash from the elasticsearch container but there is no directory /var/log/elasticsearch/… All logs should be stdout ,stderr. So I did docker logs -f elasticsearch > {“type”: “server”, “timestamp”: “2021-03-25T08:49:55,791Z”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “master node changed {previous , current [{[elastic-node3.my.domain.com](http://elastic-node3.my.domain.com)}{6QYuggRKRkueYuH\_pOfb3g}{tm0bHrK1T–rLHSXdVgM9Q}{192.168.10.216}{192.168.10.216:9300}{dimr}]}, term: 137, version: 469, reason: ApplyCommitRequest{term=137, version=469, sourceNode={[elastic-node3.my.domain.com](http://elastic-node3.my.domain.com)}{6QYuggRKRkueYuH\_pOfb3g}{tm0bHrK1T–rLHSXdVgM9Q}{192.168.10.216}{192.168.10.216:9300}{dimr}}”, “cluster.uuid”: “\_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” } > {“type”: “server”, “timestamp”: “2021-03-25T08:49:55,854Z”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “removed {{[elastic-node2.my.domain.com](http://elastic-node2.my.domain.com)}{RgOq1qp8Rxa4i1FFpHdbqA}{RcRy5QYTTqWkIHb4sFTrXg}{192.168.10.215}{192.168.10.215:9300}{dimr}}, term: 137, version: 470, reason: ApplyCommitRequest{term=137, version=470, sourceNode={[elastic-node3.my.domain.com](http://elastic-node3.my.domain.com)}{6QYuggRKRkueYuH\_pOfb3g}{tm0bHrK1T–rLHSXdVgM9Q}{192.168.10.216}{192.168.10.216:9300}{dimr}}”, “cluster.uuid”: “\_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” } > {“type”: “server”, “timestamp”: “2021-03-25T08:50:03,568Z”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “added {{[elastic-node2.my.domain.com](http://elastic-node2.my.domain.com)}{RgOq1qp8Rxa4i1FFpHdbqA}{Tfo8VZkVTLK2k8m3OAwqhA}{192.168.10.215}{192.168.10.215:9300}{dimr}}, term: 137, version: 472, reason: ApplyCommitRequest{term=137, version=472, sourceNode={[elastic-node3.my.domain.com](http://elastic-node3.my.domain.com)}{6QYuggRKRkueYuH\_pOfb3g}{tm0bHrK1T–rLHSXdVgM9Q}{192.168.10.216}{192.168.10.216:9300}{dimr}}”, “cluster.uuid”: “\_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” } > {“type”: “server”, “timestamp”: “2021-03-25T09:03:10,831Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “stopping …”, “cluster.uuid”: “\_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” } > {“type”: “server”, “timestamp”: “2021-03-25T09:03:10,837Z”, “level”: “INFO”, “component”: “o.e.c.c.Coordinator”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “master node [{[elastic-node3.my.domain.com](http://elastic-node3.my.domain.com)}{6QYuggRKRkueYuH\_pOfb3g}{tm0bHrK1T–rLHSXdVgM9Q}{192.168.10.216}{192.168.10.216:9300}{dimr}] failed, restarting discovery”, “cluster.uuid”: “\_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” , > “stacktrace”: [“org.elasticsearch.transport.NodeDisconnectedException: [[elastic-node3.my.domain.com](http://elastic-node3.my.domain.com)][192.168.10.216:9300][disconnected] disconnected”] } > {“type”: “server”, “timestamp”: “2021-03-25T09:03:10,838Z”, “level”: “DEBUG”, “component”: “o.e.d.SeedHostsResolver”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “resolveConfiguredHosts: lifecycle is STOPPED, not proceeding”, “cluster.uuid”: “\_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” } > {“type”: “server”, “timestamp”: “2021-03-25T09:03:10,844Z”, “level”: “DEBUG”, “component”: “o.e.d.PeerFinder”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “Peer{transportAddress=192.168.10.216:9300, discoveryNode=null, peersRequestInFlight=false} connection failed”, “cluster.uuid”: “\_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” , > “stacktrace”: [“java.lang.IllegalStateException: transport has been stopped”, > “at org.elasticsearch.transport.TcpTransport.ensureOpen(TcpTransport.java:845) ~[elasticsearch-7.10.2.jar:7.10.2]”, > “at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:290) ~[elasticsearch-7.10.2.jar:7.10.2]”, > “at org.elasticsearch.transport.ClusterConnectionManager.internalOpenConnection(ClusterConnectionManager.java:254) ~[elasticsearch-7.10.2.jar:7.10.2]”, > “at org.elasticsearch.transport.ClusterConnectionManager.openConnection(ClusterConnectionManager.java:95) ~[elasticsearch-7.10.2.jar:7.10.2]”, > “at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:420) ~[elasticsearch-7.10.2.jar:7.10.2]”, > “at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1.doRun(HandshakingTransportAddressConnector.java:86) [elasticsearch-7.10.2.jar:7.10.2]”, > “at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) [elasticsearch-7.10.2.jar:7.10.2]”, > “at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]”, > “at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]”, > “at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]”, > “at java.lang.Thread.run(Thread.java:832) [?:?]”] } > {“type”: “server”, “timestamp”: “2021-03-25T09:03:10,844Z”, “level”: “DEBUG”, “component”: “o.e.d.PeerFinder”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “Peer{transportAddress=192.168.10.215:9300, discoveryNode=null, peersRequestInFlight=false} connection failed”, “cluster.uuid”: “\_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” , > “stacktrace”: [“java.lang.IllegalStateException: transport has been stopped”, > “at org.elasticsearch.transport.TcpTransport.ensureOpen(TcpTransport.java:845) ~[elasticsearch-7.10.2.jar:7.10.2]”, > “at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:290) ~[elasticsearch-7.10.2.jar:7.10.2]”, > “at org.elasticsearch.transport.ClusterConnectionManager.internalOpenConnection(ClusterConnectionManager.java:254) ~[elasticsearch-7.10.2.jar:7.10.2]”, > “at org.elasticsearch.transport.ClusterConnectionManager.openConnection(ClusterConnectionManager.java:95) ~[elasticsearch-7.10.2.jar:7.10.2]”, > “at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:420) ~[elasticsearch-7.10.2.jar:7.10.2]”, > “at org.elasticsearch.discovery.HandshakingTransportAddressConnector$1.doRun(HandshakingTransportAddressConnector.java:86) [elasticsearch-7.10.2.jar:7.10.2]”, > “at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:743) [elasticsearch-7.10.2.jar:7.10.2]”, > “at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.2.jar:7.10.2]”, > “at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]”, > “at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]”, > “at java.lang.Thread.run(Thread.java:832) [?:?]”] } > {“type”: “server”, “timestamp”: “2021-03-25T09:03:11,020Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “stopped”, “cluster.uuid”: “\_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” } > {“type”: “server”, “timestamp”: “2021-03-25T09:03:11,021Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “closing …”, “cluster.uuid”: “\_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” } > {“type”: “server”, “timestamp”: “2021-03-25T09:03:11,030Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “closed”, “cluster.uuid”: “\_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” } > [INFO tini (1)] Spawned child process ‘/usr/local/bin/docker-entrypoint.sh’ with pid ‘7’ > [INFO tini (1)] Main child exited normally (with status ‘143’) > {“type”: “server”, “timestamp”: “2021-03-25T09:04:17,674Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “version[7.10.2], pid[7], build[oss/docker/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.19.0-13-amd64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:17,676Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:17,676Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-8395086767268250659, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs\_err\_pid%p.log, -Xlog:gc\*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms20480m, -Xmx20480m, -XX:MaxDirectMemorySize=10737418240, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=docker, -Des.bundled\_jdk=true]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,234Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [aggs-matrix-stats]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,234Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [analysis-common]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,235Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [geo]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,235Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [ingest-common]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,235Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [ingest-geoip]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,235Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [ingest-user-agent]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,235Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [kibana]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,236Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [lang-expression]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,236Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [lang-mustache]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,236Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [lang-painless]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,236Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [mapper-extras]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,236Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [parent-join]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,237Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [percolator]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,237Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [rank-eval]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,237Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [reindex]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,237Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [repository-url]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,237Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “loaded module [transport-netty4]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,238Z”, “level”: “INFO”, “component”: “o.e.p.PluginsService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “no plugins loaded” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,274Z”, “level”: “INFO”, “component”: “o.e.e.NodeEnvironment”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “using [1] data paths, mounts [[/usr/share/elasticsearch/data (:/volume1/elasticsearch/node1)]], net usable\_space [46.8tb], net total\_space [46.8tb], types [nfs]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,275Z”, “level”: “INFO”, “component”: “o.e.e.NodeEnvironment”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “heap size [20gb], compressed ordinary object pointers [true]” } > {“type”: “deprecation”, “timestamp”: “2021-03-25T09:04:18,298Z”, “level”: “DEPRECATION”, “component”: “o.e.d.c.s.Settings”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “[node.master] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.” } > {“type”: “deprecation”, “timestamp”: “2021-03-25T09:04:18,304Z”, “level”: “DEPRECATION”, “component”: “o.e.d.c.s.Settings”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “[node.data] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:18,377Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “node name [[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)], node ID [TptkW61sT96vQOAWvI4Qmg], cluster name [graylog], roles [master, remote\_cluster\_client, data, ingest]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:19,632Z”, “level”: “DEBUG”, “component”: “o.e.d.z.ElectMasterService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “using minimum\_master\_nodes [-1]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:20,541Z”, “level”: “INFO”, “component”: “o.e.t.NettyAllocator”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “creating NettyAllocator with the following configs: [name=elasticsearch\_configured, chunk\_size=1mb, suggested\_max\_allocation\_size=1mb, factors={es.unsafe.use\_netty\_default\_chunk\_and\_page\_size=false, g1gc\_enabled=true, g1gc\_region\_size=16mb}]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:20,574Z”, “level”: “DEBUG”, “component”: “o.e.d.SettingsBasedSeedHostsProvider”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “using initial hosts [[elastic-node2.my.domain.com](http://elastic-node2.my.domain.com), [elastic-node3.my.domain.com](http://elastic-node3.my.domain.com)]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:20,595Z”, “level”: “INFO”, “component”: “o.e.d.DiscoveryModule”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “using discovery type [zen] and seed hosts providers [settings]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:20,793Z”, “level”: “WARN”, “component”: “o.e.g.DanglingIndicesState”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “gateway.auto\_import\_dangling\_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:20,910Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “initialized” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:20,910Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “starting …” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:20,984Z”, “level”: “INFO”, “component”: “o.e.t.TransportService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “publish\_address {192.168.10.214:9300}, bound\_addresses {0.0.0.0:9300}” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:21,143Z”, “level”: “INFO”, “component”: “o.e.b.BootstrapChecks”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “bound or publishing to a non-loopback address, enforcing bootstrap checks” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:21,144Z”, “level”: “DEBUG”, “component”: “o.e.d.SeedHostsResolver”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “using max\_concurrent\_resolvers [10], resolver timeout [5s]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:21,145Z”, “level”: “INFO”, “component”: “o.e.c.c.Coordinator”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “cluster UUID [\_WWVJSF9S4KCVXcTUR7Smg]” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:21,357Z”, “level”: “INFO”, “component”: “o.e.c.s.ClusterApplierService”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “master node changed {previous , current [{[elastic-node2.my.domain.com](http://elastic-node2.my.domain.com)}{RgOq1qp8Rxa4i1FFpHdbqA}{-WKkKPbaQJ-qjOdwqQ\_tVw}{192.168.10.215}{192.168.10.215:9300}{dimr}]}, added {{[elastic-node2.my.domain.com](http://elastic-node2.my.domain.com)}{RgOq1qp8Rxa4i1FFpHdbqA}{-WKkKPbaQJ-qjOdwqQ\_tVw}{192.168.10.215}{192.168.10.215:9300}{dimr},{[elastic-node3.my.domain.com](http://elastic-node3.my.domain.com)}{6QYuggRKRkueYuH\_pOfb3g}{2mVQSIFSTHiIHbskfLhbzA}{192.168.10.216}{192.168.10.216:9300}{dimr}}, term: 138, version: 483, reason: ApplyCommitRequest{term=138, version=483, sourceNode={[elastic-node2.my.domain.com](http://elastic-node2.my.domain.com)}{RgOq1qp8Rxa4i1FFpHdbqA}{-WKkKPbaQJ-qjOdwqQ\_tVw}{192.168.10.215}{192.168.10.215:9300}{dimr}}” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:21,382Z”, “level”: “INFO”, “component”: “o.e.h.AbstractHttpServerTransport”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “publish\_address {192.168.16.3:9200}, bound\_addresses {0.0.0.0:9200}”, “cluster.uuid”: “\_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” } > {“type”: “server”, “timestamp”: “2021-03-25T09:04:21,383Z”, “level”: “INFO”, “component”: “o.e.n.Node”, “cluster.name”: “graylog”, “node.name”: “[elastic-node1.my.domain.com](http://elastic-node1.my.domain.com)”, “message”: “started”, “cluster.uuid”: “\_WWVJSF9S4KCVXcTUR7Smg”, “node.id”: “TptkW61sT96vQOAWvI4Qmg” } ---
**Author:** ![hollowdew](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@hollowdew](https://community.graylog.org/u/hollowdew)\ **Post date:** [March 25, 2021, 11:48am UTC](https://community.graylog.org/t/graylog-doesnt-process-messages-anymore-els-cluster-is-green/19252/4 "2021-03-25T11:48:00Z")
I already tried: - shutdown graylog nodes → delete graylog\_journal - restart all graylog nodes and elasticsearch nodes - graceful shutdown both graylog nodes via WebUI - stop processing → start processing on both nodes - stop all inputs → start all inputs ---
**Author:** ![hollowdew](https://avatars.discourse-cdn.com/v4/letter/h/4bbf92/32.png) [@hollowdew](https://community.graylog.org/u/hollowdew)\ **Post date:** [March 26, 2021, 8:04am UTC](https://community.graylog.org/t/graylog-doesnt-process-messages-anymore-els-cluster-is-green/19252/5 "2021-03-26T08:04:02Z")
Finally graylog processing messages. I just exported and deleted all extractors from all inputs. I only use GROK Extractors. Where can I debug that? On of the Extractors put all graylog nodes to freeze and no processing messages. ---
**Author:** ![cawfehman](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/cawfehman/32/4132_2.png) [@cawfehman](https://community.graylog.org/u/cawfehman)\ **Post date:** [March 29, 2021, 4:58pm UTC](https://community.graylog.org/t/graylog-doesnt-process-messages-anymore-els-cluster-is-green/19252/6 "2021-03-29T16:58:53Z")
might’ve been something similar to this thread and this idea. > [@Graylog has stopped storing messages in ES - no errors in log](https://community.graylog.org/t/graylog-has-stopped-storing-messages-in-es-no-errors-in-log/19181/3): > > It could be like the GROK-lock issues I was seeing - next time it locks up check the process-buffer dump on the node? ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [April 12, 2021, 4:59pm UTC](https://community.graylog.org/t/graylog-doesnt-process-messages-anymore-els-cluster-is-green/19252/7 "2021-04-12T16:59:22Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.