Garylog and Elasticsearch disks errors

[Anthonyjmm20 (Anthony)]

I logged into my Graylogserver and I noticed that my network logs didn’t show, so I decided to check the server notifications and watched this three:

Elasticsearch nodes disk usage above flood stage watermark

Elasticsearch nodes disk usage above high watermark

Elasticsearch nodes disk usage above low watermark

I’m using a Ubuntu 20.04.4 LTS, the graylog ver 4.1.13 the elastic search ver is

curl -XGET ‘http://localhost:9200’
{
“name” : “speedtest”,
“cluster_name” : “graylog”,
“cluster_uuid” : “xJKU6FDqQVKM_-EEv9AFmw”,
“version” : {
"number" : “7.10.2”,
“build_flavor” : “oss”,
“build_type” : “deb”,
“build_hash” : “747e1cc71def077253878a59143c1f785afa92b9”,
“build_date” : “2021-01-13T00:42:12.435326Z”,
“build_snapshot” : false,
“lucene_version” : “8.7.0”,
“minimum_wire_compatibility_version” : “6.8.0”,
“minimum_index_compatibility_version” : “6.0.0-beta1”
},
“tagline” : “You Know, for Search”

I read about change the elasticsearch.yml config:

cluster.routing.allocation.disk.watermark.low: 85%,
cluster.routing.allocation.disk.watermark.high: 90%,
cluster.routing.allocation.disk.watermark.flood_stage: 95%,

but when I did that i didn’t work and the service didn’t start

I’m not sure about how I can solve this or where I can erase the older logfiles so the server can continue working

Also a Iwant to increase the graylog log space

Thank u for the help

[cawfehman]

what’s your disk utilization look like?

df -ah

Here’s a good overview of this and some steps to take

Disk Watermarks In Elasticsearch

Add log space is achieved by either adding more storage to your existing ES nodes, or adding more nodes.

[Anthonyjmm20 (Anthony)]

this is the disk utilization

This server has 2HDD but i don recognize the second HDD (assuming that dev/sda2/ is the first disk)

How can I add more space to my ES node (I only use one) or where I can find the olders files data? to erase them

[cawfehman]

What is your index rotation strategy? Are you set to delete, close, archive, do nothing?

The safest way to do it would be through Graylog. Go to System | Indices. Select the Index you want to delete from. I assume you’d want to delete the oldest ones first, so scroll down the list and click on actions | delete index.

as for adding storage to you environment, not much I can offer there since your environment is just that, your environment. best to simply google that.