Periodical stream interruption

Graylog Central (peer support)
1

Hi everybody,

l’m currently facing something really strange, every monday, when i get back to work, it looks like the stream flow is interrupted.

There are few errors into my graylog server log :

ERROR [IndexRotationThread] Couldn’t point deflector to a new index
org.graylog2.indexer.ElasticsearchException: Couldn’t check existence of alias zimbra_deflector

ERROR [IndexRetentionThread] Uncaught exception in periodical
org.graylog2.indexer.ElasticsearchException: Couldn’t collect indices for alias graylog_7_reopened

ERROR [IndexRotationThread] Couldn’t point deflector to a new index
org.graylog2.indexer.ElasticsearchException: Couldn’t collect aliases for index pattern graylog_*

ERROR [AESTools] Could not decrypt value.
javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

Looking to the shards :

curl -X GET 10.10.15.107:9200/_cat_shards
{“error”:{“root_cause”:[{“type”:“illegal_argument_exception”,“reason”:“No endpoint or operation is available at [_cat_shards]”}],“type”:“illegal_argument_exception”,“reason”:“No endpoint or operation is available at [_cat_shards]”},“status”:400}

Looking to the indices :

curl 10.10.15.107:9200/_cat/indices?v

everything is green and open.

I know that rebooting the server will solve my problem, but the idea is to identify what’s missing
instead of an auto reboot.

Any idea? please…

2

you should really check your ES logfiles - you will get something out of them. Like low disk space or similar.

3

I’ve forgotten to add some details about it. There is no problem around low disk space and the elasticsearch graylog.log is almost empty of any error. (there is no /var/log/elasticsearch/graylog-2019-02-24.log for yesterday).

For the /var/log/elasticsearch/graylog-2019-02-23.log, some warns during ten minutes :
exception caught on transport layer [[id: 0x1c326412, L:/10.10.15.107:9300 - R:/10.10.15.109:48584]], closing connection

but no error…

4

curl -X GET 10.10.15.107:9200/_cat_shards

You may want to try _cat/shards instead :wink:

5

That’s right, and all is started with the correct command

So i’ve still no idea of the problem but all the streams are still empty since saturday morning.

6

Any idea for my issue…?

7

Have you checked if one of the indices is missing?
As in xyz_14 and xyz_16 are present, but xyz_15 is not? ( -> Recalculate index ranges)

Any unscheduled reboots? (or scheduled reboots, that graylog / elasticsearch are unaware of…)

8

There is not scheduled or unscheduled reboot. The more surprising thing is that the web server and every mechanism on it is running without error.

I’ve no missing indice neither…

9

did you run this on a vcenter that does balance the machines automatically based on load?

10

All the host settings are set as fix settings, there is no dynamic config for ressources or load balancing…

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.