# Graylog blocked

**URL:** <https://community.graylog.org/t/graylog-blocked/17795>\
**Category:** Graylog Central (peer support)\
**Created:** [November 10, 2020, 8:17am UTC](https://community.graylog.org/t/graylog-blocked/17795 "2020-11-10T08:17:45Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Labidi](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/labidi/32/5597_2.png) [@Labidi](https://community.graylog.org/u/Labidi)\
**Post date:** [November 10, 2020, 8:17am UTC](https://community.graylog.org/t/graylog-blocked/17795/1 "2020-11-10T08:17:45Z")

</div>

Hello ,  
Suddenly my graylog is crashed, I cannot receive the palo alto firewall logs or no others Vm linux for my infrastructure - Please help

 ![log blocked](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/0/03deff27b77574e6cd28b3af4513ca6e2924ce92.png)

 ![urgent](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/0/0c9c2b664bb3bf64a1907aeda7a9a04cbb654edd.png)

 ![fail show log](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/b/bb0eff4144764937c1051001b4101a15b63d19c6.png)  
 ![unable widget](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/5/584e3e938eede12572bbb86413b4aac478a46965.png)

 ![jobs](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/4/4bf096dfc30c97d34da75b830ced3e80a42c9a8a.png)

---

<div class="post-metadata">

**Author:** ![Labidi](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/labidi/32/5597_2.png) [@Labidi](https://community.graylog.org/u/Labidi)\
**Post date:** [November 10, 2020, 9:08am UTC](https://community.graylog.org/t/graylog-blocked/17795/2 "2020-11-10T09:08:38Z")

</div>

**There were 6,858 failed indexing attempts in the last 24 hours.**

---

<div class="post-metadata">

**Author:** ![cawfehman](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/cawfehman/32/4132_2.png) [@cawfehman](https://community.graylog.org/u/cawfehman)\
**Post date:** [November 10, 2020, 4:16pm UTC](https://community.graylog.org/t/graylog-blocked/17795/3 "2020-11-10T16:16:08Z")

</div>

I’m a little confused by your screenshots since one has your ES status as green and another has it as red. Is your ES status red or green? Have you looked at the ES logs?

---

<div class="post-metadata">

**Author:** ![Labidi](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/labidi/32/5597_2.png) [@Labidi](https://community.graylog.org/u/Labidi)\
**Post date:** [November 10, 2020, 4:38pm UTC](https://community.graylog.org/t/graylog-blocked/17795/4 "2020-11-10T16:38:39Z")

</div>

I think the problem is in the indexing of the logs, the indexing posed a problem 😕 **There were 6,858 failed indexing attempts in the last 24 hours.**

now I can receive the logs , but the problem in I **ndexer failues**  
Display Dhashboard:  
**While retrieving data for this widget, the following error(s) occurred:**

- Unable to perform search query: java.util.concurrent.ExecutionException: ElasticsearchException[java.util.concurrent.ExecutionException: CircuitBreakingException[[parent] Data too large, data for [source] would be [728827392/695mb], which is larger than the limit of [726571417/692.9mb], usages [request=0/0b, fielddata=149837342/142.8mb, in\_flight\_requests=932292/910.4kb, accounting=578057758/551.2mb]]]; nested: ExecutionException[CircuitBreakingException[[parent] Data too large, data for [source] would be [728827392/695mb], which is larger than the limit of [726571417/692.9mb], usages [request=0/0b, fielddata=149837342/142.8mb, in\_flight\_requests=932292/910.4kb, accounting=578057758/551.2mb]]]; nested: CircuitBreakingException[[parent] Data too large, data for [source] would be [728827392/695mb], which is larger than the limit of [726571417/692.9mb], usages [request=0/0b, fielddata=149837342/142.8mb, in\_flight\_requests=932292/910.4kb, accounting=578057758/551.2mb]];.

---

<div class="post-metadata">

**Author:** ![cawfehman](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/cawfehman/32/4132_2.png) [@cawfehman](https://community.graylog.org/u/cawfehman)\
**Post date:** [November 10, 2020, 4:51pm UTC](https://community.graylog.org/t/graylog-blocked/17795/5 "2020-11-10T16:51:41Z")

</div>

Looks like the issue is with ES and the amount of memory/heap you have allocated.

> **[What does this error mean - Data too large, data for \[\]](https://discuss.elastic.co/t/what-does-this-error-mean-data-too-large-data-for-transport-request/209345)**
>
> Hello, please explain me what this error mean ? And what I need to do to get rid of it : \[2019-11-25T00:20:58,521\]\[WARN \]\[o.e.x.m.e.l.LocalExporter\] \[host\] unexpected error while indexing monitoring document...

what version of ES are you running? what are your system specs?

---

<div class="post-metadata">

**Author:** ![Labidi](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/labidi/32/5597_2.png) [@Labidi](https://community.graylog.org/u/Labidi)\
**Post date:** [November 11, 2020, 8:13am UTC](https://community.graylog.org/t/graylog-blocked/17795/6 "2020-11-11T08:13:37Z")

</div>

![elasticsearch problem](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/0/0e89608058bedfd1ba9ea235f994e3b3efac1b79.png)

**Journal utilization is too high and may go over the limit soon. Please verify that your Elasticsearch cluster is healthy and fast enough. You may also want to review your Graylog journal settings and set a higher limit. (Node: _9c349e50-cda8-43ca-b6c1-a980b10fd5b6_ )**

I couldn’t solve this problem , please Help

---

<div class="post-metadata">

**Author:** ![Labidi](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/labidi/32/5597_2.png) [@Labidi](https://community.graylog.org/u/Labidi)\
**Post date:** [November 11, 2020, 9:34am UTC](https://community.graylog.org/t/graylog-blocked/17795/7 "2020-11-11T09:34:40Z")

</div>

@cawfehman I thnik the problem **in Journal utilization is too high (triggered an hour ago)**

**Journal utilization is too high and may go over the limit soon. Please verify that your Elasticsearch cluster is healthy and fast enough. You may also want to review your Graylog journal settings and set a higher limit. (Node: _9c349e50-cda8-43ca-b6c1-a980b10fd5b6_ )**

---

<div class="post-metadata">

**Author:** ![Labidi](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/labidi/32/5597_2.png) [@Labidi](https://community.graylog.org/u/Labidi)\
**Post date:** [November 11, 2020, 1:16pm UTC](https://community.graylog.org/t/graylog-blocked/17795/8 "2020-11-11T13:16:29Z")

</div>

Good for now, I solved the problem by modifying :

message\_journal\_max\_age  
message\_journal\_max\_size

in the graylog-server.conf file

---

<div class="post-metadata">

**Author:** ![cawfehman](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/cawfehman/32/4132_2.png) [@cawfehman](https://community.graylog.org/u/cawfehman)\
**Post date:** [November 11, 2020, 3:06pm UTC](https://community.graylog.org/t/graylog-blocked/17795/9 "2020-11-11T15:06:13Z")

</div>

glad it’s working… your journal fills up when Graylog can’t push the messages through the processing pipelines and out to ES fast enough. There’s alot of potential bottlenecks, from disk to cpu/RAM. If it happens again (I would think it would) you may need to dig a bit more.

Good luck.

---

<div class="post-metadata">

**Author:** ![Labidi](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/labidi/32/5597_2.png) [@Labidi](https://community.graylog.org/u/Labidi)\
**Post date:** [November 11, 2020, 3:24pm UTC](https://community.graylog.org/t/graylog-blocked/17795/10 "2020-11-11T15:24:51Z")

</div>

thank you very much for your feedback, it’s very nice, good, my graylog server I think it’s quite efficient on the resources side (ram, cpu, storage …) … but the problem on the firewall side, I really receive a lot of logs.

@cawfehman Good luck to you too 🙏

---

<div class="post-metadata">

**Author:** ![Labidi](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/labidi/32/5597_2.png) [@Labidi](https://community.graylog.org/u/Labidi)\
**Post date:** [November 11, 2020, 4:28pm UTC](https://community.graylog.org/t/graylog-blocked/17795/11 "2020-11-11T16:28:37Z")

</div>

@cawfehman

The problem is back, up there in the dashboard for sending log: out always 0 (in / out) (843/0)  
**While retrieving data for this widget, the following error(s) occurred:**

- Connection refused (Connection refused).

 ![out 0](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/9/966fd23db2001a6968f087f4b6e2f598678a6f5e.png)

---

<div class="post-metadata">

**Author:** ![Labidi](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/labidi/32/5597_2.png) [@Labidi](https://community.graylog.org/u/Labidi)\
**Post date:** [November 11, 2020, 4:35pm UTC](https://community.graylog.org/t/graylog-blocked/17795/12 "2020-11-11T16:35:17Z")

</div>

![journal](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/2/2b6967c3748779c5ebb2fc127fb434ef4c8ec6e8.png)

---

<div class="post-metadata">

**Author:** ![cawfehman](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/cawfehman/32/4132_2.png) [@cawfehman](https://community.graylog.org/u/cawfehman)\
**Post date:** [November 11, 2020, 7:19pm UTC](https://community.graylog.org/t/graylog-blocked/17795/13 "2020-11-11T19:19:20Z")

</div>

what do your buffers look like? input, process, output buffers? What’s your java heap at? How much CPU/RAM do you have?

---

<div class="post-metadata">

**Author:** ![Labidi](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/labidi/32/5597_2.png) [@Labidi](https://community.graylog.org/u/Labidi)\
**Post date:** [November 12, 2020, 7:54am UTC](https://community.graylog.org/t/graylog-blocked/17795/14 "2020-11-12T07:54:05Z")

</div>

Hi @cawfehman my graylog server have : 16GB RAM, CPU: 8 CPU

 ![elasticsearch problem](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/0/0e89608058bedfd1ba9ea235f994e3b3efac1b79.png)

---

<div class="post-metadata">

**Author:** ![Labidi](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/labidi/32/5597_2.png) [@Labidi](https://community.graylog.org/u/Labidi)\
**Post date:** [November 12, 2020, 7:54am UTC](https://community.graylog.org/t/graylog-blocked/17795/15 "2020-11-12T07:54:34Z")

</div>

![problem elasticsearch](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/5/57d62d62b6f0aee3916589ba79dbabeb55a4f121.png)

---

<div class="post-metadata">

**Author:** ![Labidi](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/labidi/32/5597_2.png) [@Labidi](https://community.graylog.org/u/Labidi)\
**Post date:** [November 16, 2020, 4:27pm UTC](https://community.graylog.org/t/graylog-blocked/17795/16 "2020-11-16T16:27:46Z")

</div>

![problem agin](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/e/ef3e7ce753574939c3810cdda50830f5544436ed.png)

**While retrieving data for this widget, the following error(s) occurred:**

- Connection refused (Connection refused).

---

<div class="post-metadata">

**Author:** ![cawfehman](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/cawfehman/32/4132_2.png) [@cawfehman](https://community.graylog.org/u/cawfehman)\
**Post date:** [November 18, 2020, 2:23pm UTC](https://community.graylog.org/t/graylog-blocked/17795/17 "2020-11-18T14:23:17Z")

</div>

is your elasticsearch and you graylog on the same system?

how are your processors allocated in the graylog server.conf?  
 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/7/73f0d634bfabf06441e62632c1da6e63ef2a8bb0.png)

how much Java heap do you have allocated?

> **[Frequently asked questions — Graylog 3.3.8 documentation](https://docs.graylog.org/en/3.3/pages/faq.html?highlight=java#raise-the-java-heap)**

---

<div class="post-metadata">

**Author:** ![Labidi](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/labidi/32/5597_2.png) [@Labidi](https://community.graylog.org/u/Labidi)\
**Post date:** [November 18, 2020, 6:41pm UTC](https://community.graylog.org/t/graylog-blocked/17795/18 "2020-11-18T18:41:59Z")

</div>

Hello @cawfehman  
Yes Graylog and elasticsearch in the same host ! this is the current state of processbuffer\_processors and outputbuffer\_processors:

**processbuffer\_processors = 5**  
**outputbuffer\_processors = 3**  
How much does the value increase ??? How can I know , how much Java heap allocated ???

---

<div class="post-metadata">

**Author:** ![Labidi](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/labidi/32/5597_2.png) [@Labidi](https://community.graylog.org/u/Labidi)\
**Post date:** [November 18, 2020, 6:55pm UTC](https://community.graylog.org/t/graylog-blocked/17795/19 "2020-11-18T18:55:59Z")

</div>

![problem](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/d/d2ae8b34a89f4303c723dce4d7028aaa155cd0be.png)

---

<div class="post-metadata">

**Author:** ![ttsandrew](https://avatars.discourse-cdn.com/v4/letter/t/97f17d/32.png) [@ttsandrew](https://community.graylog.org/u/ttsandrew)\
**Post date:** [November 18, 2020, 6:56pm UTC](https://community.graylog.org/t/graylog-blocked/17795/20 "2020-11-18T18:56:20Z")

</div>

@Labidi, is this the same server we discussed in this thread?

> [@Problem elasticsearch](https://community.graylog.org/t/problem-elasticsearch/17873/3):
>
> Hello @ttsandrew Thanks for your feedback, ES is reachable by Graylog and is green I need a solution. the problem that I receive a lot of logs

You mention there that the specs are 12 GB RAM and 8 CPU, but here you say 16 GB and 8 CPU. To help you allocate memory optimally can you clarify?

If this is the same server being discussed in both threads it’s apparent Elasticsearch is having some issues, in the other thread you did not mention that the Elasticsearch service was having periods of being unreachable which means most likely that it is crashing repeatedly. What errors are present in the Elasticsearch log?

You can view heap size in /etc/elasticsearch/jvm.options

[Next page](https://community.graylog.org/t/graylog-blocked/17795.md?page=2)
