# Journal utilization 99% has gone over 95% **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [April 27, 2020, 7:18pm UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167 "2020-04-27T19:18:08Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![sadman](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@sadman](https://community.graylog.org/u/sadman)\ **Post date:** [April 27, 2020, 7:18pm UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/1 "2020-04-27T19:18:08Z")
**I am having this problem regularly. This affects the processing of messages that stop being processed.** I have a strategy to rotate indexes by size (I also tried each time) to keep 2 indexes, but I realize that when the graylog needs to rotate the third one and delete the oldest one, it gets stuck and, therefore, the messages stop processing. How can I improve the performance of this? I have 32 GB of RAM, 16 GB for Elasticsearch and 8 for Graylog’s JVM. Last messages in “server.log” Graylog > _2020-04-27T16:01:51.508-03:00 WARN [KafkaJournal] Journal utilization (99.0%) has gone over 95%._ > _\> 2020-04-27T16:02:51.459-03:00 WARN [KafkaJournal] Journal utilization (99.0%) has gone over 95%._ > _\> 2020-04-27T16:03:21.944-03:00 INFO [AbstractIndexCountBasedRetentionStrategy] Number of indices (3) higher than limit (2). Running retention for 1 indices._ > _\> 2020-04-27T16:03:21.947-03:00 INFO [AbstractIndexCountBasedRetentionStrategy] Running retention strategy [org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy] for index \_ > _\> 2020-04-27T16:03:23.730-03:00 INFO [DeletionRetentionStrategy] Finished index retention strategy [delete] for index \ in 1782ms._ > _\> 2020-04-27T16:03:31.940-03:00 INFO [AbstractRotationStrategy] Deflector index \ (index set \) should be rotated, Pointing deflector to new index now!_ > _\> 2020-04-27T16:03:31.941-03:00 INFO [MongoIndexSet] Cycling from \ to \._ > _\> 2020-04-27T16:03:31.941-03:00 INFO [MongoIndexSet] Creating target index \._ > > ![graylog-community](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/1/1c80151d03b9e43345214870d1a7c2468ed920d5.png) ---
**Author:** ![makarands](https://avatars.discourse-cdn.com/v4/letter/m/838e76/32.png) [@makarands](https://community.graylog.org/u/makarands)\ **Post date:** [April 28, 2020, 11:42am UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/2 "2020-04-28T11:42:01Z")
@sadman: You need to check journal message directory configuration(i.e. message\_journal\_dir = " ") on Graylog configuration “server.conf”. This directory will be used to store the message journal and it must exclusively be used by Graylog and must not contain any other files than the ones created by Graylog itself. If you need more information please share message directory path and its utilization. I hope this helps you!!! ---
**Author:** ![lcosta](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/lcosta/32/6109_2.png) [@lcosta](https://community.graylog.org/u/lcosta)\ **Post date:** [April 28, 2020, 11:50am UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/3 "2020-04-28T11:50:25Z")
Journal has a especifc folder in the system, check “server.conf” It´s not recomend to change the folder location after the first install, you can change the limits for the journal. Check the doc: First here:[http://docs.graylog.org/en/2.4/pages/faq.html#what-does-journal-utilization-is-too-high-mean](http://docs.graylog.org/en/2.4/pages/faq.html#what-does-journal-utilization-is-too-high-mean) Then for the configuration file: [http://docs.graylog.org/en/2.4/pages/configuration/server.conf.html#output-batch-size](http://docs.graylog.org/en/2.4/pages/configuration/server.conf.html#output-batch-size) ---
**Author:** ![sadman](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@sadman](https://community.graylog.org/u/sadman)\ **Post date:** [April 29, 2020, 4:42am UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/4 "2020-04-29T04:42:31Z")
Thank you very much for the contributions, I will check and forward updates here. ---
**Author:** ![sadman](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@sadman](https://community.graylog.org/u/sadman)\ **Post date:** [May 11, 2020, 1:31pm UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/5 "2020-05-11T13:31:27Z")
We have Graylog with the following configurations: > ring\_size= 65536 ( # of messages in each buffer) > inputbuffer\_ring\_size=65536 > > processbuffer\_processors = 5 > outputbuffer\_processors = 3 > > output\_batch\_size = 1000 > journal\_age = 15 min > journal\_size = 15 gb **GRAYLOG RAM = 8gb** **ELASTICSEARCH RAM =16gb** Only 1 node, Only 1 machine We are getting like 25,000 messages per second and outputs only 5,000 messages per second The journal is filling up quickly and getting an error “Journal Utilization is too high” so, can anyone please help me in calculating the increase of output\_processors and output\_batch\_size ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [May 12, 2020, 4:06pm UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/6 "2020-05-12T16:06:52Z")
for that amount of messages it looks like your processing power in terms of cpu cycles is not enough. ---
**Author:** ![sadman](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@sadman](https://community.graylog.org/u/sadman)\ **Post date:** [May 14, 2020, 2:46am UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/7 "2020-05-14T02:46:22Z")
@jan Sorry I didn’t send the complete information. The system runs a single node in a VM that has 24vCPU In total resources, I have: 24CPU and 32 GB of RAM I divided 50% for elasticsearch (16GB RAM) and 25% for Graylog (8 GB RAM), the remaining 25% for SO. ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [May 14, 2020, 2:01pm UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/8 "2020-05-14T14:01:59Z")
he @sadman did you limit the cores that are usable by elasticsearch in the elasticsearch configuration? If not Graylog and elasticsearch are fighting for the available cores. In addition with that ingest rate special during index rotation you have not enough computing ressources and the graylog journal is used to buffer. You might want to raise the size of the journal and/or add additional compute power. ---
**Author:** ![sadman](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@sadman](https://community.graylog.org/u/sadman)\ **Post date:** [May 14, 2020, 5:39pm UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/9 "2020-05-14T17:39:53Z")
> [@jan](#): > > size of the journal How do I limit the cores usable by Elasticsearch and Graylog? I limited the JVM’s memory to 16 EL and 8 GR. As for the size of the journal, what do you recommend? The default value is configured. I believe that this is exactly the behavior, using the journal above what can and start receiving unprocessed messages, and this increases when need to rotate the indexes ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [May 15, 2020, 9:23am UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/10 "2020-05-15T09:23:05Z")
he @sadman read the docs: [https://www.elastic.co/guide/en/elasticsearch/reference/6.8/modules-threadpool.html#processors](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/modules-threadpool.html#processors) the journal should have the size that fits to your needs. If you get paged when your elasticsearch is dead and you can fix that in 4 hours, the journal should have the size to cover this period of time. If you do not get paged and elasticsearch can die on friday noon and you notice it earliest monday morning you might need a journal that can cover 3-4 days of logs. But you need to have this disk space exclusive for graylog. Cause the journal will get damaged if the configured size is not available and you loose all messages in the journal. ---
**Author:** ![sadman](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@sadman](https://community.graylog.org/u/sadman)\ **Post date:** [May 26, 2020, 8:43pm UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/11 "2020-05-26T20:43:42Z")
I did not find in the configuration file “_ **server.conf** _” or in “_ **elasticsearch.yml** _” the option to configure the processors. It would be the option of **“inputbuffer\_processors”** in “server.conf”??? ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [May 27, 2020, 6:39am UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/12 "2020-05-27T06:39:06Z")
it is the graylog server.conf … [https://docs.graylog.org/en/3.3/pages/configuration/server.conf.html](https://docs.graylog.org/en/3.3/pages/configuration/server.conf.html) check for `buffer_pro` to find all options for processing, input and ouput. ---
**Author:** ![sadman](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@sadman](https://community.graylog.org/u/sadman)\ **Post date:** [May 27, 2020, 4:16pm UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/13 "2020-05-27T16:16:57Z")
I have 24 CPU. do you recommend any value where i can move? ---
**Author:** ![sadman](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@sadman](https://community.graylog.org/u/sadman)\ **Post date:** [May 27, 2020, 4:33pm UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/14 "2020-05-27T16:33:00Z")
configured this way for now and kept message processing stable > [@](#): > > processbuffer\_processors = 12 > outputbuffer\_processors = 7 > inputbuffer\_processors = 5 ---
**Author:** ![jan](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jan/32/11_2.png) [@jan](https://community.graylog.org/u/jan)\ **Post date:** [May 28, 2020, 6:36am UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/15 "2020-05-28T06:36:23Z")
he @sadman I would go with 2 for input, 3 for output and the processing to 16. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [June 11, 2020, 6:36am UTC](https://community.graylog.org/t/journal-utilization-99-has-gone-over-95/15167/16 "2020-06-11T06:36:31Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.