# No more messages flowing inbound? Started over twice now... what am I doing wrong? **URL:** \ **Category:** Graylog Central (peer support)\ **Tags:** elastic\ **Created:** [February 1, 2022, 10:19pm UTC](https://community.graylog.org/t/no-more-messages-flowing-inbound-started-over-twice-now-what-am-i-doing-wrong/22540 "2022-02-01T22:19:05Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![gleep52](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gleep52/32/10843_2.png) [@gleep52](https://community.graylog.org/u/gleep52)\ **Post date:** [February 1, 2022, 10:19pm UTC](https://community.graylog.org/t/no-more-messages-flowing-inbound-started-over-twice-now-what-am-i-doing-wrong/22540/1 "2022-02-01T22:19:05Z")
Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question. **Don’t forget to select tags to help index your topic!** **1. Describe your incident:** Installed ubuntu 20 and followed the guide to install graylog - set up three DCs and our Meraki gear to forward syslog traffic to port 1515. Also tinkered with nxlog forwarding with a beats input on port 5044. Everything works well for a day or two and then messages stop flowing inbound. **2. Describe your environment:** - OS Information: Ubuntu 20.04 on hyper-v VM 8 cores Xeon Gold 6148 CPU 24GB memory - Package Version: ii elasticsearch-oss 7.10.2 amd64 Distributed RESTful search engine built for the cloud ii graylog-4.2-repository 1-4 all Package to install Graylog 4.2 GPG key and repository ii graylog-integrations-plugins 4.2.5-1 all Graylog Integrations plugins ii graylog-server 4.2.5-1 all Graylog server ii mongodb-org 4.0.28 amd64 MongoDB open source document-oriented database system (metapackage) ii mongodb-org-mongos 4.0.28 amd64 MongoDB sharded cluster query router ii mongodb-org-server 4.0.28 amd64 MongoDB database server ii mongodb-org-shell 4.0.28 amd64 MongoDB shell client ii mongodb-org-tools 4.0.28 amd64 MongoDB tools - Service logs, configurations, and environment variables: server.conf file: is\_master = true node\_id\_file = /etc/graylog/server/node-id password\_secret = root\_password\_sha2 = bin\_dir = /usr/share/graylog-server/bin data\_dir = /var/lib/graylog-server plugin\_dir = /usr/share/graylog-server/plugin http\_bind\_address = 10.10.10.27:9000 http\_enable\_cors = false rotation\_strategy = count elasticsearch\_max\_docs\_per\_index = 20000000 elasticsearch\_max\_number\_of\_indices = 20 retention\_strategy = delete elasticsearch\_shards = 4 elasticsearch\_replicas = 0 elasticsearch\_index\_prefix = graylog allow\_leading\_wildcard\_searches = false allow\_highlighting = false elasticsearch\_analyzer = standard output\_batch\_size = 500 output\_flush\_interval = 1 output\_fault\_count\_threshold = 5 output\_fault\_penalty\_seconds = 30 processbuffer\_processors = 5 outputbuffer\_processors = 3 processor\_wait\_strategy = blocking ring\_size = 65536 inputbuffer\_ring\_size = 65536 inputbuffer\_processors = 2 inputbuffer\_wait\_strategy = blocking message\_journal\_enabled = true message\_journal\_dir = /var/lib/graylog-server/journal lb\_recognition\_period\_seconds = 3 mongodb\_uri = mongodb://localhost/graylog mongodb\_max\_connections = 1000 mongodb\_threads\_allowed\_to\_block\_multiplier = 5 proxied\_requests\_thread\_pool\_size = 32 **3. What steps have you already taken to try and solve the problem?** tail -f /var/log/graylog-server/server.log 2022-02-01T15:38:10.404-06:00 WARN [LocalKafkaJournal] Journal utilization (101.0%) has gone over 95%. 2022-02-01T15:38:41.843-06:00 INFO [connection] Opened connection [connectionId{localValue:17, serverValue:17}] to localhost:27017 2022-02-01T15:38:41.850-06:00 INFO [connection] Opened connection [connectionId{localValue:15, serverValue:15}] to localhost:27017 2022-02-01T15:38:41.851-06:00 INFO [connection] Opened connection [connectionId{localValue:16, serverValue:16}] to localhost:27017 2022-02-01T15:38:41.851-06:00 INFO [connection] Opened connection [connectionId{localValue:14, serverValue:12}] to localhost:27017 2022-02-01T15:38:41.852-06:00 INFO [connection] Opened connection [connectionId{localValue:13, serverValue:13}] to localhost:27017 2022-02-01T15:38:41.854-06:00 INFO [connection] Opened connection [connectionId{localValue:11, serverValue:11}] to localhost:27017 2022-02-01T15:38:41.857-06:00 INFO [connection] Opened connection [connectionId{localValue:12, serverValue:14}] to localhost:27017 2022-02-01T15:39:11.056-06:00 WARN [LocalKafkaJournal] Journal utilization (98.0%) has gone over 95%. 2022-02-01T15:39:11.058-06:00 INFO [LocalKafkaJournal] Journal usage is 98.00% (threshold 100%), changing load balancer status from THROTTLED to ALIVE _Journal usage overrun maybe?_ curl -XGET [http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true) { “cluster\_name” : “graylog”, “status” : “green”, “timed\_out” : false, “number\_of\_nodes” : 1, “number\_of\_data\_nodes” : 1, “active\_primary\_shards” : 20, “active\_shards” : 20, “relocating\_shards” : 0, “initializing\_shards” : 0, “unassigned\_shards” : 0, “delayed\_unassigned\_shards” : 0, “number\_of\_pending\_tasks” : 0, “number\_of\_in\_flight\_fetch” : 0, “task\_max\_waiting\_in\_queue\_millis” : 0, “active\_shards\_percent\_as\_number” : 100.0 } I’m not sure what a shard is - but assuming if it’s 100% then it’s full or too busy to process more. What can I do to get messages flowing again? And most importantly please help me understand why this happened to prevent it from reoccurring. (Like deleting a folder of queued messages is nice to know how and what to do, but why didn’t they process and what settings do I need to adjust for proper retention and automatic digesting)? **4. How can the community help?** I’m guessing my elasticsearch settings are wrong or perhaps the server isn’t powerful enough? I cannot get into the /etc/elasticsearch directory as access is denied - is that expected? I don’t want to start modifying file permissions without understanding why or what that might break. How can I get messages flowing again? **Helpful Posting Tips:** [Tips for Posting Questions that Get Answers](https://community.graylog.org/t/tips-for-posting-questions-that-get-answers/21828) [Hold down CTRL and link on link to open tips documents in a separate tab] ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [February 1, 2022, 11:33pm UTC](https://community.graylog.org/t/no-more-messages-flowing-inbound-started-over-twice-now-what-am-i-doing-wrong/22540/2 "2022-02-01T23:33:14Z")
Hello && Welcome > [@gleep52](#): > > 2022-02-01T15:38:10.404-06:00 WARN [LocalKafkaJournal] Journal utilization (101.0%) has gone over 95%. This warning mean that your Journal utilization full. It could be a couple different things. 1. Check your Elasticsearch status (systemctl status elasticsearch) and ES log files. 2. Perhaps Increase your Journal from default of 5 Gb to maybe 10 GB IF you have enough room on your Drive to do so. 3. If your process buffer are shown full on the Web UI you may want to increase that, they would be in your server.conf file \*\*processbuffer\_processors = 5 \*\* and wait. This may take time to complete and the logs in your journal. How much resource you have and/or how much messages GL is ingesting. Your process buffer is the big hitter, then your output buffer. If buffers are full, I would only increase the Process buffer to 6 and wait to see if it goes down. Take note when you GL Volume is full, or your journal is full, or your buffers are at 100% logs coming in will pause. You will be unable to see those messages until your issue is resolved. Normally when the journal is full this means the resource are unable to keep up with the amount of messages coming in or Elasticsearch issue. > [@gleep52](#): > > get into the /etc/elasticsearch directory as access is denied - is that expected? So the user you logged into Graylog server does not have the permission to that directory. So this seam like a permission issue also. Check your Server logs Elasticsearch/Graylog. EDIT: Since your using nxlog I had a problem a while back one Device was sending over 12,000 messages. This was from an nxlog client and it filled my journal up really quick over tim you may want to check your shippers. ---
**Author:** ![gleep52](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gleep52/32/10843_2.png) [@gleep52](https://community.graylog.org/u/gleep52)\ **Post date:** [February 1, 2022, 11:59pm UTC](https://community.graylog.org/t/no-more-messages-flowing-inbound-started-over-twice-now-what-am-i-doing-wrong/22540/3 "2022-02-01T23:59:55Z")
The user I am logged in with is the user I installed graylog with but as I am not able to see inside the /etc/elasticsearch/ folder - I’m guessing this might be an issue? Maybe once the journal fills up it cannot write them to elastic? Or am I miss understanding the flow? Is it journal → elastic → mongo? Status is up and running but I’m not sure how to access the logs since I can’t see in the directory? Is it best to chmod the elasticsearch folder or better to give my user root access, even if temporarily? Messages do seem to skyrocket when I first opened the ports for ingestion (4000/second messages) but eventually slowed down to 150/second or so… No idea what graylog is capable of or if I’m choking it with data or not. Seeing as I’m using ssh to do everything - what are good ways to check the journal size, hard drive space, etc? How do I increase the journal size? The server has either 300 or 600gb for its storage. What happens to old logs - assuming it deletes by default per the server config - but is there a way to archive them onto an SMB share or such instead for longevity? ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [February 2, 2022, 12:11am UTC](https://community.graylog.org/t/no-more-messages-flowing-inbound-started-over-twice-now-what-am-i-doing-wrong/22540/4 "2022-02-02T00:11:36Z")
> [@gleep52](#): > > I am not able to see inside the /etc/elasticsearch/ folder - I’m guessing this might be an issue Yes its probably combination of a few issue you may have. If the user you installed Graylog with has root permission then you could adjust it, but if the user you install Graylog doesn’t not have permission to /etc/elasticsearch directory you may want to talk to the local admin about getting access. > [@gleep52](#): > > what are good ways to check the journal size journal is in the Web UI under _System/Node_ > [@gleep52](#): > > hard drive space, etc? `root # df -h` > [@gleep52](#): > > How do I increase the journal size? message\_journal\_max\_size = 5gb > [@gleep52](#): > > What happens to old logs - assuming it deletes by default per the server config - but is there a way to archive them onto an SMB share or such instead for longevity? You have a lot going on. I would first find out WHY you journal is filling up over it capacity and resolve that issue. Also… > [@gsmith](#): > > Check your Server logs Elasticsearch/Graylog. If you don’t have access to log files then you have more problems ---
**Author:** ![bking](https://avatars.discourse-cdn.com/v4/letter/b/dc4da7/32.png) [@bking](https://community.graylog.org/u/bking)\ **Post date:** [February 10, 2022, 6:37pm UTC](https://community.graylog.org/t/no-more-messages-flowing-inbound-started-over-twice-now-what-am-i-doing-wrong/22540/5 "2022-02-10T18:37:02Z")
I might have some insight regarding your (in)ability to access certain files and directories in Ubuntu. I’m not a Linux person, so it’s possible I’m violating best practice here. Please take it with a grain of salt. I’ve noticed that on Ubuntu server (I’m assuming you installed Ubuntu 20.04 LTS), the ‘administrative’ user you make as part of the install isn’t actually a root user (which I think is on purpose?). You can ‘sudo’ to do administrative commands, and edit files that require root access (e.g. “sudo nano /etc/graylog/server/server.conf”), but you can’t actually browse those directories as the admin user. This is where I may be violating best practice, but if you do “sudo su” you’ll have access to a root command prompt, which will let you ‘cd’ and ‘ls’ any directory (including /etc/elasticsearch). I’m pretty sure that’s all operating as intended, and it may not actually be a problem if your normal admin user can’t directly access those directories. I hope that helps (and that I haven’t lead you astray). ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [February 24, 2022, 6:37pm UTC](https://community.graylog.org/t/no-more-messages-flowing-inbound-started-over-twice-now-what-am-i-doing-wrong/22540/6 "2022-02-24T18:37:24Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.