# Graylog stopped with timeout **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [November 23, 2021, 11:53am UTC](https://community.graylog.org/t/graylog-stopped-with-timeout/21834 "2021-11-23T11:53:56Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Sharzad](https://avatars.discourse-cdn.com/v4/letter/s/35a633/32.png) [@Sharzad](https://community.graylog.org/u/Sharzad)\ **Post date:** [November 23, 2021, 11:53am UTC](https://community.graylog.org/t/graylog-stopped-with-timeout/21834/1 "2021-11-23T11:53:56Z")
i have graylog 4.x on ubuntu 20.04. standard installation. I can log in to my gralog but all data is one week old. services says running but : graylog-server.service: Failed with result ‘timeout’. in graylog log i have warning WARN [MessagesAdapterES6] Failed to index message: index=\ id= error=\<{“type”:“cluster\_block\_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}\> in syslog i have error : 2021-11-18T23:25:02.093084+00:00 rsyslog systemd[1]: motd-news.service: Failed with result ‘exit-code’. 2021-11-18T23:25:02.094389+00:00 rsyslog systemd[1]: Failed to start Message of the Day. ---
**Author:** ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)\ **Post date:** [November 23, 2021, 12:17pm UTC](https://community.graylog.org/t/graylog-stopped-with-timeout/21834/2 "2021-11-23T12:17:27Z")
> [@Sharzad](#): > > blocked by: [FORBIDDEN/12/index read-only / allow delete This usually means you have hit the highwater mark on the disk where your Elasticsearch keeps it’s data. You haven’t posted which versions of Elasticsearch you have but this [Elasticsearch article](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/disk-allocator.html) should give you the information you need about Elasticsearch disk thresholds and how to set/clear them. You will need to either roll of data from indexes or increase disk space before you clear the read-only on the graylog\_0 index. ---
**Author:** ![tellistone](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tellistone/32/10200_2.png) [@tellistone](https://community.graylog.org/u/tellistone)\ **Post date:** [November 23, 2021, 12:24pm UTC](https://community.graylog.org/t/graylog-stopped-with-timeout/21834/3 "2021-11-23T12:24:25Z")
If you can attach your Graylog and Elasticsearch logs we can offer more specific advice, but as above it’s likely you’re running out of ES storage. ---
**Author:** ![Sharzad](https://avatars.discourse-cdn.com/v4/letter/s/35a633/32.png) [@Sharzad](https://community.graylog.org/u/Sharzad)\ **Post date:** [November 23, 2021, 12:49pm UTC](https://community.graylog.org/t/graylog-stopped-with-timeout/21834/4 "2021-11-23T12:49:47Z")
my elasticsearch is : ~# curl localhost:9200 { “name” : “peoR6Gc”, “cluster\_name” : “graylog”, “cluster\_uuid” : “o-clJDFLSz2BcV-fK5ld\_Q”, “version” : { _ **"number" : "6.8.20"** _, “build\_flavor” : “oss”, “build\_type” : “deb”, “build\_hash” : “c859302”, “build\_date” : “2021-10-07T22:00:24.085009Z”, “build\_snapshot” : false, “lucene\_version” : “7.7.3”, “minimum\_wire\_compatibility\_version” : “5.6.0”, “minimum\_index\_compatibility\_version” : “5.0.0” }, “tagline” : “You Know, for Search” and graylog version 4.2 ---
**Author:** ![Sharzad](https://avatars.discourse-cdn.com/v4/letter/s/35a633/32.png) [@Sharzad](https://community.graylog.org/u/Sharzad)\ **Post date:** [November 23, 2021, 1:03pm UTC](https://community.graylog.org/t/graylog-stopped-with-timeout/21834/5 "2021-11-23T13:03:25Z")
logfile /var/log/elas…/graylog say [2021-11-19T22:23:35,163][WARN][o.e.c.r.a.DiskThresholdMonitor] [peoR6Gc] high disk watermark [90%] exceeded on [peoR6GcRQpqhJZlebPSo5g][peoR6Gc][/var/lib/elasticsearch/nodes/0] free: 1.4gb[9.3%], shards will be relocated away from this node … … ---
**Author:** ![Sharzad](https://avatars.discourse-cdn.com/v4/letter/s/35a633/32.png) [@Sharzad](https://community.graylog.org/u/Sharzad)\ **Post date:** [November 23, 2021, 1:05pm UTC](https://community.graylog.org/t/graylog-stopped-with-timeout/21834/6 "2021-11-23T13:05:54Z")
… [2021-11-20T03:45:38,447][INFO][o.e.c.r.a.DiskThresholdMonitor] [peoR6Gc] low disk watermark [85%] exceeded on [peoR6GcRQpqhJZlebPSo5g][peoR6Gc][/var/lib/elasticsearch/nodes/0] free: 2.3gb[14.8%], replicas will not be assigned to this node ---
**Author:** ![Sharzad](https://avatars.discourse-cdn.com/v4/letter/s/35a633/32.png) [@Sharzad](https://community.graylog.org/u/Sharzad)\ **Post date:** [November 23, 2021, 7:40pm UTC](https://community.graylog.org/t/graylog-stopped-with-timeout/21834/7 "2021-11-23T19:40:59Z")
@tmacgbay , how to increase disk space. i have 12 gig disk space ---
**Author:** ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)\ **Post date:** [November 24, 2021, 4:02pm UTC](https://community.graylog.org/t/graylog-stopped-with-timeout/21834/8 "2021-11-24T16:02:33Z")
Increasing your disk space goes beyond Graylog… I know little about what you have and how you have it set up. I am sure there are plenty of posts on the internet that you could Google and find that are relevant to your system. ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [November 24, 2021, 10:38pm UTC](https://community.graylog.org/t/graylog-stopped-with-timeout/21834/9 "2021-11-24T22:38:06Z")
> [@Sharzad](#): > > how to increase disk space. i have 12 gig disk space This depends on if your Graylog server resides on physical hardware or if its on a virtual machine. If its hardware either you need a new HDD and clone you graylog server to the larger drive, if your current dive has more space you can extend the portion. If your Graylog server is on a virtual machine it easy to add more space to the drive. Once you increase the volume you then need to add it to the correct portion on the Graylog server. [https://help.ubuntu.com/stable/ubuntu-help/disk-resize.html.en](https://help.ubuntu.com/stable/ubuntu-help/disk-resize.html.en) Hope that helps ---
**Author:** ![Sharzad](https://avatars.discourse-cdn.com/v4/letter/s/35a633/32.png) [@Sharzad](https://community.graylog.org/u/Sharzad)\ **Post date:** [November 25, 2021, 7:38am UTC](https://community.graylog.org/t/graylog-stopped-with-timeout/21834/10 "2021-11-25T07:38:01Z")
i know how to resize disk but my issue is there is 12 gig back on my ubuntu, why they say no disk ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [November 25, 2021, 10:15pm UTC](https://community.graylog.org/t/graylog-stopped-with-timeout/21834/11 "2021-11-25T22:15:06Z")
Hello, My apologies, I misread your post. > [@Sharzad](#): > > my issue is there is 12 gig back on my ubuntu, why they say no disk - I believe Its because of this statement in your logs. ```auto low disk watermark [85%] exceeded on ``` - Definition of low disk watermark [85%] exceeded. > When disk usage on a host hits 85 percent, the Elasticsearch service prevents shard allocation and **stops working**. This disk usage threshold is an Elasticsearch configuration. **By default** , the **cluster.routing.allocation.disk.watermark.low watermark is set to 85%** to prevent Elasticsearch from **allocating new shards** to hosts once **disk usage on the host exceeds 85 percent.** - The Next stage is High Disk Watermark > The next level is the High Disk Watermark stage. The default value is 95%. Elasticsearch enforces a read-only index block (index.blocks.read\_only\_allow\_delete) on every index that has one or more shards allocated on the node, and that has at least one disk exceeding the flood stage. This setting is the last resort to prevent nodes from running out of disk space. Depending on your Elasticsearch version, the release mechanism is different: - Before Elasticsearch 7, the index block must be released manually when the disk utilization falls below the high watermark. - Since Elasticsearch 7, the index block is automatically released when the disk utilization falls below the high watermark. - Maybe these post will enlighten you on what’s going on. > **[Elasticsearch Low Disk Watermark - How to Resolve and Avoid it](https://opster.com/guides/elasticsearch/capacity-planning/elasticsearch-low-disk-watermark/)** > > Low disk watermark is one of the various thresholds on your Elasticsearch cluster. Here are possible actions you can take to resolve... > **[Elasticsearch High Disk Watermark- Understanding Watermark Thresholds](https://opster.com/guides/elasticsearch/capacity-planning/elasticsearch-high-disk-watermark/)** > > High disk watermark is one of the various thresholds on your Elasticsearch cluster. Passing this threshold is a warning and you should not... Hope that helps ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [December 9, 2021, 10:15pm UTC](https://community.graylog.org/t/graylog-stopped-with-timeout/21834/12 "2021-12-09T22:15:18Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.