# Slow down the Graylog system and not all inputs are process **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [March 16, 2017, 5:31am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482 "2017-03-16T05:31:15Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![tharu85](https://avatars.discourse-cdn.com/v4/letter/t/e95f7d/32.png) [@tharu85](https://community.graylog.org/u/tharu85)\ **Post date:** [March 16, 2017, 5:31am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/1 "2017-03-16T05:31:15Z")
Due to performance and JVM issue, upgraded to Graylog 2.2.2 from 2.0.x. After upgraded Graylog worked 5 days perfectly and smoothly. suddenly system becomes slow down. Now the WEB GUI, takes considerable amount of time to load each pages and not all inputs logs are processing. In source area it is showing only 3 input sources logs were processed. But earlier it shows all input sources. (See below log history) ![](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/1X/a7ef36c0203b9e4bda8367d7ad2e5f57ba7a21d7.jpg) ![](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/1X/cf0ae029f4d8c16fd47a90b240bde6bb158f9b67.jpg) Some times in input segment it is showing inputs are not running, after 5-10 seconds it’ll automatically running. (see the below images as an example for single input) ![](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/1X/16575d46e2acb97e9d10334d3208729b26f9ad10.jpg) ![](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/1X/3c0c65edab65bafd9b615136a78f276a4e7a1241.jpg) ![](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/1X/fc112e30ec4ced7242977094c8574b99bfbdc1d6.jpg) The system is having 32GB of RAM. Although I have increased the message\_journal\_max\_size to 5gb But I couldn’t rectify the issue. As per my understanding and I guess there is one point that Graylog unable to handle due to it’s memory issue. Can anyone face these kind of issues ?. I need to resolve this issue asap. Please share me the solution if any one has. ---
**Author:** ![tharu85](https://avatars.discourse-cdn.com/v4/letter/t/e95f7d/32.png) [@tharu85](https://community.graylog.org/u/tharu85)\ **Post date:** [March 16, 2017, 5:45am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/2 "2017-03-16T05:45:03Z")
Refer the server log. It shows that input nodes timed outs ![](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/1X/43be3bb4d0baa69db8d75a3e4d883104f9f9eea1.jpg) Note : I can’t go for fresh installation or restore the snap since existing Graylog server has valid date. ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [March 16, 2017, 8:26am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/3 "2017-03-16T08:26:02Z")
Hej Tharaka, the given information are little short as you did not include some kind of information what you had tried to resolve the issue. Additional without any log files we can only guess what might be the problem in your situation. What did you find in the log files at the time where the processing drops? What actions did you perform latest on the system before the drop? Did you check if some other changes (network, routing, firewall) might have impact on the number of sources? regards Jan ---
**Author:** ![tharu85](https://avatars.discourse-cdn.com/v4/letter/t/e95f7d/32.png) [@tharu85](https://community.graylog.org/u/tharu85)\ **Post date:** [March 16, 2017, 8:33am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/4 "2017-03-16T08:33:41Z")
Hey Jan. This system is worked perfectly since August 2016. In January 2017 the system performance getting dropped. In the /var/log/graylog-server/server.log it always shows that OpenJDK JVM issue. So that I have upgraded into latest Graylog version on last Friday 10th March 2017. After the upgrade system worked perfectly and very smoothly. During that time no error observed. But Since day before yesterday the system got slowdown and the inputs are not working properly. All the input going down and going up time to time. For further clarification, what kind of a log source are you required. Any how I need to sort out this issue ASAP. Please help me to fix it. ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [March 16, 2017, 9:05am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/5 "2017-03-16T09:05:47Z")
Hej, the logfile from that day when slowdown was noticed will be of interest. But what you described sound that you are at the limits of your hardware. Maybe you might want to look at [this blog post about tuning](https://www.graylog.org/blog/74-back-to-basics-from-single-server-to-graylog-cluster) and [how to monitor graylog](https://www.graylog.org/blog/86-back-to-basics-monitoring-graylog). regards Jan ---
**Author:** ![tharu85](https://avatars.discourse-cdn.com/v4/letter/t/e95f7d/32.png) [@tharu85](https://community.graylog.org/u/tharu85)\ **Post date:** [March 16, 2017, 9:24am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/6 "2017-03-16T09:24:40Z")
Hi Jay, It can’t be with Hardware. Because in my system it is used 32 VCPUs and 32GB of RAM. 1 TB of HDD and only 270GBs are used. So I don’t think the issue with H/W. This might be with Graylog inputs. Please let me know how to attached the log file here. ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [March 16, 2017, 10:25am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/7 "2017-03-16T10:25:02Z")
you can just use the upload function of the editor or you put the log elsewhere in the net and place a link. ---
**Author:** ![tharu85](https://avatars.discourse-cdn.com/v4/letter/t/e95f7d/32.png) [@tharu85](https://community.graylog.org/u/tharu85)\ **Post date:** [March 16, 2017, 10:46am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/8 "2017-03-16T10:46:45Z")
Hi Jay. I have uploaded the file to a 3rd party server and the link is shared here with decryption key. This is server.log file (/var/log/graylog-server/server.log) [https://mega.nz/#!yZRH2bLI](https://mega.nz/#!yZRH2bLI) Deykryption Key “!P2\_eXjbuOYEtkqOT4kRobfWhOO92ZnP24Bqny3gklWs” ---
**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\ **Post date:** [March 16, 2017, 11:22am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/9 "2017-03-16T11:22:39Z")
How did you install Graylog? How did you configure Graylog and its JVM settings? What’s running on the system, just Graylog or also Elasticsearch etc.? ---
**Author:** ![tharu85](https://avatars.discourse-cdn.com/v4/letter/t/e95f7d/32.png) [@tharu85](https://community.graylog.org/u/tharu85)\ **Post date:** [March 16, 2017, 11:34am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/10 "2017-03-16T11:34:33Z")
@jochen This VM is used only for Graylog Server with Elasticsearch (Log analyzer) OS : CentOS 6 (2.6.32-642.el6.x86\_64) Packages: Elasticsearch 2.3.3 Graylog 2.2.2 MongoDB 3.2.7-1.el6.x86\_64 OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) Packages installed using RPM System is a VM and H/W specifications are 32 vCPUs 32 GB of RAM 1 TB of HDD free -h total used free shared buffers cached Mem: 31G 15G 15G 204K 197M 7.4G -/+ buffers/cache: 8.0G 23G Swap: 8.0G 0B 8.0G Apart from that Kibana and logstash installed in this system ---
**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\ **Post date:** [March 16, 2017, 11:47am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/11 "2017-03-16T11:47:29Z")
You can clearly see that your system is using 100% swap which is pretty much the worst case. How did you configure all relevant components on that machine? What are your JVM settings for Graylog and Elasticsearch? ---
**Author:** ![tharu85](https://avatars.discourse-cdn.com/v4/letter/t/e95f7d/32.png) [@tharu85](https://community.graylog.org/u/tharu85)\ **Post date:** [March 16, 2017, 11:47am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/12 "2017-03-16T11:47:40Z")
@jochen This Graylog system worked perfectly since August 2016 to January 2017 with Graylog 2.0.1 version and after some memory issue upgraded to version to 2.2.2 to 10th March 2017. After that system again worked perfectly till 14th March 2017 and getting system slowdown. ---
**Author:** ![tharu85](https://avatars.discourse-cdn.com/v4/letter/t/e95f7d/32.png) [@tharu85](https://community.graylog.org/u/tharu85)\ **Post date:** [March 16, 2017, 11:49am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/13 "2017-03-16T11:49:53Z")
@jochen Swap: 8.0G 0B 8.0G As you can see, free space is [8.GB](http://8.GB) for swap. 0B used and 8GB are free All necessary settings were configured from the beginning and issue raised recently. ---
**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\ **Post date:** [March 16, 2017, 12:03pm UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/14 "2017-03-16T12:03:26Z")
> [@tharu85](#): > > As you can see, free space is 8.GB for swap. 0B used and 8GB are free True, I misread the output of `free`. > [@tharu85](#): > > All necessary settings were configured from the beginning and issue raised recently. That’s no answer to my question. ---
**Author:** ![tharu85](https://avatars.discourse-cdn.com/v4/letter/t/e95f7d/32.png) [@tharu85](https://community.graylog.org/u/tharu85)\ **Post date:** [March 16, 2017, 12:07pm UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/15 "2017-03-16T12:07:52Z")
@jochen Fine. I’ll glad to give you the details. Let me know what are the exact information that you are required ? ---
**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\ **Post date:** [March 16, 2017, 12:34pm UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/16 "2017-03-16T12:34:15Z")
Configuration of Graylog and Elasticsearch, including their JVM settings. ---
**Author:** ![tharu85](https://avatars.discourse-cdn.com/v4/letter/t/e95f7d/32.png) [@tharu85](https://community.graylog.org/u/tharu85)\ **Post date:** [March 16, 2017, 12:48pm UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/17 "2017-03-16T12:48:19Z")
Refer the attachment for following files. /etc/sysconfig/graylog-server /etc/sysconfig/elasticsearch /etc/graylog/server/server.conf /etc/elasticsearch/elasticsearch.yml download link [https://mega.nz/#!2ZBkVLyZ](https://mega.nz/#!2ZBkVLyZ) Decryption key ‘’!-3-grrzWvcYyUux2HX7CLi\_vJW1Rk9we5DOjIg6k\_0o" ---
**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\ **Post date:** [March 16, 2017, 1:22pm UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/18 "2017-03-16T13:22:24Z")
The `ES_HEAP_SIZE` setting is very low with 2 gigabytes. Try using at least 8 gigabytes for Elasticsearch. Also make sure to check the logs of your Elasticsearch node(s). ---
**Author:** ![tharu85](https://avatars.discourse-cdn.com/v4/letter/t/e95f7d/32.png) [@tharu85](https://community.graylog.org/u/tharu85)\ **Post date:** [March 17, 2017, 6:00am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/19 "2017-03-17T06:00:46Z")
I have increased the ES\_HEAP\_SIZE in to 8g and monitored for an hour. Nothing happened and issue is still persist ---
**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\ **Post date:** [March 17, 2017, 8:13am UTC](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482/20 "2017-03-17T08:13:06Z")
Please post the logs of all relevant components (Graylog, Elasticsearch, etc.). [Next page](https://community.graylog.org/t/slow-down-the-graylog-system-and-not-all-inputs-are-process/482.md?page=2)