Sources Page - Could not load sources data - after Upgrade Graylog and Elasticsearch

Andreas · August 25, 2017, 2:18pm

Hi,

after upgrading Graylog to 2.3 and Elasticsearch from 2.3 to 5.5 the sources page did not display the sources.
Elasticsearch log shows following error:

[2017-08-25T16:09:54,801][DEBUG][o.e.a.s.TransportSearchAction] [elasticsearch-n15-01] All shards failed for phase: [query]
org.elasticsearch.ElasticsearchException$1: Fielddata is disabled on text fields by default. Set fielddata=true on [source] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
        at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:618) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:125) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:240) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:88) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:47) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.action.search.InitialSearchPhase$1.onFailure(InitialSearchPhase.java:156) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1067) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.transport.TcpTransport.lambda$handleException$16(TcpTransport.java:1467) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110) [elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:1465) [elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:1457) [elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1401) [elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-5.5.2.jar:5.5.2]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]

How can I enable field data or change ‘source’ to keyword type?

Thanks,

Andreas

jochen · August 25, 2017, 2:54pm

fielddata = true is already used for the source field in the index template created by Graylog:

github.com

Graylog2/graylog2-server/blob/2.3.1/graylog2-server/src/main/java/org/graylog2/indexer/IndexMapping5.java#L39


public class IndexMapping5 extends IndexMapping {
@Override
protected Map<String, Map<String, Object>> fieldProperties(String analyzer) {
    return ImmutableMap.of(
            "message", analyzedString(analyzer, false),
            "full_message", analyzedString(analyzer, false),
            // http://joda-time.sourceforge.net/api-release/org/joda/time/format/DateTimeFormat.html
            // http://www.elasticsearch.org/guide/reference/mapping/date-format.html
            "timestamp", typeTimeWithMillis(),
            // to support wildcard searches in source we need to lowercase the content (wildcard search lowercases search term)
            "source", analyzedString("analyzer_keyword", true),
            "streams", notAnalyzedString());
}


@Override
protected Map<String, Object> notAnalyzedString() {
    return ImmutableMap.of("type", "keyword");
}


private Map<String, Object> analyzedString(String analyzer, boolean fieldData) {
    return ImmutableMap.of(

Make sure that this index template is being used and that your indices (created after the upgrade to Elasticsearch 5.5.x) have the correct mapping:

Andreas · August 28, 2017, 8:10am

HI,
the template shows the correct mapping:

          "source" : {
            "fielddata" : true,
            "analyzer" : "analyzer_keyword",
            "type" : "text"
          },

I had to be patient until Graylog/Elasticsearch created a new index with the correct mapping at least.Could it be possible, that the field type of ‘source’ has been changed from string to text but the fielddata wasn’t set during the upgrade of the open index?

  "graylog_34" : {
    "mappings" : {
      "message" : {
        "dynamic_templates" : [
          {
            "internal_fields" : {
              "match" : "gl2_*",
              "mapping" : {
                "index" : "not_analyzed",
                "type" : "string"
...
          "source" : {
            "type" : "string",
            "analyzer" : "analyzer_keyword"
          },
...
  "graylog_35" : {
    "mappings" : {
      "message" : {
        "dynamic_templates" : [
          {
            "internal_fields" : {
              "match" : "gl2_*",
              "mapping" : {
                "index" : "not_analyzed",
                "type" : "string"
...
          "source" : {
            "type" : "text",
            "analyzer" : "analyzer_keyword"
          },
...
  "graylog_36" : {
    "mappings" : {
      "message" : {
        "dynamic_templates" : [
          {
            "internal_fields" : {
              "match" : "gl2_*",
              "mapping" : {
                "index" : "not_analyzed",
                "type" : "string"
...
          "source" : {
            "type" : "text",
            "analyzer" : "analyzer_keyword"
          },
...
  "graylog_37" : {
    "mappings" : {
      "message" : {
        "dynamic_templates" : [
          {
            "internal_fields" : {
              "match" : "gl2_*",
              "mapping" : {
                "type" : "keyword"
...
          "source" : {
            "type" : "text",
            "analyzer" : "analyzer_keyword",
            "fielddata" : true
          },

Regards,

Andreas

jochen · August 28, 2017, 11:47am

Yes, the field type changes from string to keyword or text (depending on some mapping attributes) when using Elasticsearch 5.x.
Also see Mapping changes | Elasticsearch Reference [5.5] | Elastic

Existing indices will not be upgraded automatically (at least not by Graylog).

If you think this is a bug, please open an issue at Issues · Graylog2/graylog2-server · GitHub

system · September 11, 2017, 11:47am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Troubkle upgrading from 3 to 4 Graylog Tech Challenges	13	779	August 9, 2021
Fielddata automatically enabled for "source" field Development	12	2854	March 11, 2020
ES fielddata java exception has broken graylog Graylog Central (peer support)	8	1047	May 14, 2019
Issues after upgrading to 3.0.0 Graylog Central (peer support)	7	1259	March 27, 2019
Indexer Failures after upgrade to 2.3.1 Graylog Central (peer support)	3	720	September 12, 2017

Sources Page - Could not load sources data - after Upgrade Graylog and Elasticsearch

Related topics