Hi there, I’m running Graylog 3.0.2 (1 master + 2 slave nodes) and 7 Elasticsearch 6.5.4 nodes (3 master +4 data nodes). I’ve noticed strange warnings and errors in /var/log/elasticsearch/graylog.log on one of my Elasticsearch data nodes:
Warning:
“[WARN ][o.e.i.c.IndicesClusterStateService] [elasticsearch-data-3] [[graylog_1400][0]] marking and sending shard failed due to [shard failure, reason [refresh failed source[schedule]]]
java.io.IOException: Bad file descriptor
at sun.nio.ch.FileDispatcherImpl.size0(Native Method) ~[?:?]
at sun.nio.ch.FileDispatcherImpl.size(FileDispatcherImpl.java:90) ~[?:?]
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:383) ~[?:?]”
Errror:
"Suppressed: java.io.IOException: Bad file descriptor
at java.io.FileDescriptor.close0(Native Method) ~[?:?]
at java.io.FileDescriptor.close(FileDescriptor.java:297) ~[?:?]
at java.io.FileDescriptor$1.close(FileDescriptor.java:88) ~[?:?]
at sun.nio.ch.FileChannelImpl$Closer.run(FileChannelImpl.java:106) ~[?:?]
at jdk.internal.ref.CleanerImpl$PhantomCleanableRef.performCleanup(CleanerImpl.java:186) ~[?:?]
at jdk.internal.ref.PhantomCleanable.clean(PhantomCleanable.java:133) ~[?:?]
at sun.nio.ch.FileChannelImpl.implCloseChannel(FileChannelImpl.java:198) ~[?:?]
Warning:
“[WARN ][o.e.i.IndexService ] [elasticsearch-data-3] [graylog_1400] failed to run task refresh - suppressing re-occurring exceptions unless the exception changes
org.elasticsearch.index.engine.RefreshFailedEngineException: Refresh failed”
Only one data node has this errors, haven’t noticed them on other data nodes
Should I be worried about there errors, what could cause them?
P.S. Already checked data disk on that VM using fsck - no errors.