Node running but indices failed to fetch


(Sai Aditya Varma) #1

Logged in to Graylog web to find default index not set. In the logs: ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
2018-04-30_05:40:53.31946 java.lang.IllegalStateException: No default index set configured. This is a bug!

I didn’t make any changes. Full logs here: https://pastebin.com/wL35mSRX


(Jan Doberstein) #2

What Graylog Version did you run? What Elasticsearch Version did you run? Is this error after first start? How did you installed Graylog?


(Sai Aditya Varma) #3

No this error is not after first start, i have been using this for a month with multiple streams and indices. I used a community AMI in AWS.

Graylog Version:
2.4.3+2c41897, codename Wildwuchs

ElasticSearch: “version” : {
“number” : “5.6.3”,
“build_hash” : “1a2f265”,
“build_date” : “2017-10-06T20:33:39.012Z”,
“build_snapshot” : false,
“lucene_version” : “6.6.1”
},

MongoDB: 3.4.9

Also i see the following in the system messages:

2018-04-30T01:40:47-04:00 908753fe / graylog Cycled index alias <indice_deflector> from to <indice_0>.
2018-04-30T01:40:47-04:00 908753fe / graylog There is no index target to point to. Creating one now.
2018-04-30T01:39:57-04:00 908753fe / graylog Cycled index alias <log_deflector> from to <log_0>.
2018-04-30T01:39:57-04:00 908753fe / graylog There is no index target to point to. Creating one now.
2018-04-30T01:38:17-04:00 908753fe / graylog Cycled index alias <set_deflector> from to <set_0>.
2018-04-30T01:38:17-04:00 908753fe / graylog There is no index target to point to. Creating one now.

(Jan Doberstein) #4

what about the elasticseach in your setup?

Did you checked the logfiles? What is the cluster health and status?

curl -X GET "localhost:9200/_cluster/health"

(Sai Aditya Varma) #5

Elasticsearch logs:

[o.e.t.n.Netty4Transport ] [2MXMvOB] exception caught on transport layer [[id: 0x5284ea6f, L:/172.31.43.119:9300 ! R:/185.143.223.154:2370]], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (3,0,0,2f)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:392) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:359) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.logging.LoggingHandler.channelInactive(LoggingHandler.java:167) [netty-handler-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1329) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:908) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:744) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.13.Final.jar:4.1.13.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) [netty-common-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (3,0,0,2f)
at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1276) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:36) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?]
… 24 more

Cluster Health:

{
“cluster_name”:“graylog”,
“status”:“yellow”,
“timed_out”:false,
“number_of_nodes”:1,
“number_of_data_nodes”:1,
“active_primary_shards”:24,
“active_shards”:24,
“relocating_shards”:0,
“initializing_shards”:0,
“unassigned_shards”:4,
“delayed_unassigned_shards”:0,
“number_of_pending_tasks”:0,
“number_of_in_flight_fetch”:0,
“task_max_waiting_in_queue_millis”:0,
“active_shards_percent_as_number”:85.71428571428571
}


(Jan Doberstein) #6

first, please format your posts to have them easier to read for others. Our FAQ gives some hints on how to do that.

You have 4 unassigned shards, making the cluster yellow ( http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#cluster-status-explained ). As a wild guess - did you checkd the available space for Elasticsearch? Did you find something like low-water-mark in the elasticsearch logfiles?

When you fixed the Elasticsearch unassigned shards, your problem might be resolved. This might help you with that: https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/


(Sai Aditya Varma) #7

So after reading the article, i understood not to have a single node elasticsearch cluster. Is that true?


(system) #8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.