Unhealthy Cluster Version: 2.4.5 after disk space issue


(Rajinder Singh) #1

I am running a Graylog2(Version 2.4.5) appliance in AWS. Everything is running in 1 VM.
We did not have appropriate retention policies in place and we ran out of disk space.
mongodb did not start due to disk issues.
For us data is not important but the configuration is important.
So running graylog-ctl cleanse was not an option.
I had to manually delete data for graylog mongodb to run.
Once there was space on disk web ui came up.

I see this message:
Elasticsearch cluster is yellow. Shards: 4 active, 0 initializing, 0 relocating, 4 unassigned,

Uncommited messages deleted from journal

Journal utilization is too high

How do I assign these unassigned shards?

Commands like don’t seem to work for me.
url -X GET ‘http://localhost:9200/graylog_deflector/_mapping?pretty
curl: (7) Failed to connect to localhost port 9200: Connection refused

Our webui requires HTTPS

graylog-ctl status
run: elasticsearch: (pid 8893) 3473s; run: log: (pid 968) 475437s
run: etcd: (pid 8914) 3473s; run: log: (pid 964) 475437s
run: graylog-server: (pid 8973) 3468s; run: log: (pid 963) 475437s
run: mongodb: (pid 8993) 3468s; run: log: (pid 969) 475437s
run: nginx: (pid 9002) 3467s; run: log: (pid 965) 475437s

Here the ports that are used:
netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 10.20.10.104:9200 0.0.0.0:* LISTEN 8893/java
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 9002/nginx.conf
tcp 0 0 10.20.10.104:9300 0.0.0.0:* LISTEN 8893/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1058/sshd
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 9002/nginx.conf
tcp 0 0 0.0.0.0:27017 0.0.0.0:* LISTEN 8993/mongod
tcp 0 0 127.0.0.1:2380 0.0.0.0:* LISTEN 8914/etcd
tcp6 0 0 :::22 :::* LISTEN 1058/sshd
tcp6 0 0 :::4001 :::* LISTEN 8914/etcd
tcp6 0 0 :::9000 :::* LISTEN 8973/java
tcp6 0 0 :::12201 :::* LISTEN 8973/java
tcp6 0 0 :::2379 :::* LISTEN 8914/etcd
udp 0 0 0.0.0.0:68 0.0.0.0:* 662/dhclient
udp 0 0 10.20.10.104:123 0.0.0.0:* 1417/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0:* 1417/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:* 1417/ntpd
udp 0 0 0.0.0.0:57903 0.0.0.0:* 662/dhclient
udp6 0 0 :::12201 :::* 8973/java
udp6 0 0 :::40943 :::* 662/dhclient
udp6 0 0 fe80::108e:b9ff:fe5:123 :::* 1417/ntpd
udp6 0 0 ::1:123 :::* 1417/ntpd
udp6 0 0 :::123 :::* 1417/ntpd
udp6 0 0 :::514 :::* 8973/java

Let me know if you have any suggestions.

Thanks


(system) closed #2

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.