Loading of histogram data failed with status: Error: cannot GET (500)

Graylog Central (peer support)
1

Hi

I have a problem about histogram loading. Actually i can see the histogram but a yellow notification shows up with :
Loading of histogram data failed with status: Error: cannot GET http://192.168.2.24:9000/api/search/universal/relative/histogram?query=source%3Aexample.org&range=0&interval=day (500)

Elasticsearch is connected to Graylog.

Graylog log

2018-05-31T17:38:53.372+01:00 INFO  [ServiceManagerListener] Services are healthy
2018-05-31T17:38:53.373+01:00 INFO  [ServerBootstrap] Graylog server up and running.
2018-05-31T17:38:53.542+01:00 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2018-05-31T17:38:53.578+01:00 INFO  [InputStateListener] Input [Syslog UDP/5b0d4dd0136c5b0bc5eba756] is now STARTING
2018-05-31T17:38:53.590+01:00 INFO  [InputStateListener] Input [GELF HTTP/5b0d5e93136c5b0bc5ebb8f7] is now STARTING
2018-05-31T17:38:53.623+01:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input GELFHttpInput{title=test, type=org.graylog2.inputs.gelf.http.GELFHttpInput, nodeId=758506a3-f343-427c-9e39-03feefda4bfb} should be 1048576 but is 212992.
2018-05-31T17:38:53.628+01:00 INFO  [InputStateListener] Input [GELF HTTP/5b0d5e93136c5b0bc5ebb8f7] is now RUNNING
2018-05-31T17:38:53.635+01:00 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=rsyslog to graylog, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=758506a3-f343-427c-9e39-03feefda4bfb} should be 262144 but is 212992.
2018-05-31T17:38:53.637+01:00 INFO  [InputStateListener] Input [Syslog UDP/5b0d4dd0136c5b0bc5eba756] is now RUNNING

elasticsearch log

 at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:252) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:267) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.9.jar:5.6.9]

Thank you for your help

2

The error message from the Elasticsearch logs is incomplete. Please post the complete Elasticsearch logs.

3 
org.elasticsearch.transport.RemoteTransportException: [UaOkFFS][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Expected numeric type on field [timestamp], but got [string]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.numericField(ValuesSourceConfig.java:306) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.originalValuesSource(ValuesSourceConfig.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.toValuesSource(ValuesSourceConfig.java:246) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:51) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:225) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:226) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:55) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:252) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:267) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
[2018-06-01T10:53:25,037][DEBUG][o.e.a.s.TransportSearchAction] [UaOkFFS] All shards failed for phase: [query]
org.elasticsearch.ElasticsearchException$1: Expected numeric type on field [timestamp], but got [string]
        at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:618) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:126) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:241) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:107) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:49) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:217) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:171) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:49) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:217) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1077) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1181) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1159) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:665) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:659) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-5.6.9.jar:5.6.9]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.IllegalArgumentException: Expected numeric type on field [timestamp], but got [string]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.numericField(ValuesSourceConfig.java:306) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.originalValuesSource(ValuesSourceConfig.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.toValuesSource(ValuesSourceConfig.java:246) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:51) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:225) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:226) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:55) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:252) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:267) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.9.jar:5.6.9]
        ... 3 more
[2018-06-01T10:54:47,101][DEBUG][o.e.a.s.TransportSearchAction] [UaOkFFS] [graylog_0][0], node[UaOkFFShSiuFM4awqEIZXQ], [P], s[STARTED], a[id=sJg5AgN7TrSGcYs9CdVRxQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[graylog_0], indicesOptions=IndicesOptions[id=39, ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[message], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=5, batchedReduceSize=512, preFilterShardSize=64, source={
  "from" : 0,
  "query" : {
    "bool" : {
      "must" : [
        {
          "match_all" : {
            "boost" : 1.0
          }
        }
      ],
      "filter" : [
        {
          "bool" : {
            "must" : [
              {
                "range" : {
                  "timestamp" : {
                    "from" : "2018-06-01 08:54:47.090",
                    "to" : "2018-06-01 09:54:47.090",
                    "include_lower" : true,
                    "include_upper" : true,
                    "boost" : 1.0
                  }
                }
              }
            ],
            "disable_coord" : false,
            "adjust_pure_negative" : true,
            "boost" : 1.0
          }
        }
      ],
      "disable_coord" : false,
      "adjust_pure_negative" : true,
      "boost" : 1.0
    }
  },
  "aggregations" : {
    "gl2_histogram" : {
      "date_histogram" : {
        "field" : "timestamp",
        "interval" : "1m",
        "offset" : 0,
        "order" : {
          "_key" : "asc"
        },
        "keyed" : false,
        "min_doc_count" : 0
      }
    }
  }
}}]
org.elasticsearch.transport.RemoteTransportException: [UaOkFFS][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Expected numeric type on field [timestamp], but got [string]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.numericField(ValuesSourceConfig.java:306) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.originalValuesSource(ValuesSourceConfig.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.toValuesSource(ValuesSourceConfig.java:246) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:51) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:225) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:226) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:55) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:252) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:267) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
[2018-06-01T10:54:47,102][DEBUG][o.e.a.s.TransportSearchAction] [UaOkFFS] All shards failed for phase: [query]
org.elasticsearch.ElasticsearchException$1: Expected numeric type on field [timestamp], but got [string]
        at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:618) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:126) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:241) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:107) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:49) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:217) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:171) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:49) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:217) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1077) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1181) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1159) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:665) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:659) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-5.6.9.jar:5.6.9]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.IllegalArgumentException: Expected numeric type on field [timestamp], but got [string]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.numericField(ValuesSourceConfig.java:306) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.originalValuesSource(ValuesSourceConfig.java:289) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.toValuesSource(ValuesSourceConfig.java:246) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:51) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:225) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:226) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:55) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:252) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:267) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.9.jar:5.6.9]
        ... 3 more
4

The “timestamp” message field has to have a numeric data-type (which it has when using the Graylog default template).

Somehow, the field changed its type in one or more of the indices which you’re trying to query.

Make sure that the Graylog default index template exists in Elasticsearch and that you didn’t overwrite the mapping of the “timestamp” field. See http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html for details.

You can try rotating the current write-active index (System/Index Set/Index/Maintenance), but that won’t help with querying existing indices with the incorrect mapping for the “timestamp” field. For those, I’m afraid you’ll have to either delete them or re-index the documents (see Reindex API | Elasticsearch Reference [5.6] | Elastic).

5

Actually I followed this documentation: http://docs.graylog.org/en/2.4/pages/installation/os/ubuntu.html
So I didn’t overwrite the mapping of the “timestamp” field. I use ubuntu server 18.014 LTS .

I changed just the file /etc/elasticsearch/elasticsearch.yml and set the cluster name: cluster.name: graylog.

do you mind if I upload server.conf here ??

6

Please post the complete output of the following commands (with localhost:9200 being the address of an Elasticsearch node):

# curl -i 'http://localhost:9200/_mapping?pretty'
# curl -i 'http://localhost:9200/_template?pretty'
8 
$ curl -i 'http://localhost:9200/_mapping?pretty'
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 1250

{
  "graylog_0" : {
    "mappings" : {
      "message" : {
        "properties" : {
          "application_name" : {
            "type" : "string"
          },
          "facility" : {
            "type" : "string"
          },
          "foo" : {
            "type" : "string"
          },
          "gl2_remote_ip" : {
            "type" : "string"
          },
          "gl2_remote_port" : {
            "type" : "long"
          },
          "gl2_source_input" : {
            "type" : "string"
          },
          "gl2_source_node" : {
            "type" : "string"
          },
          "level" : {
            "type" : "long"
          },
          "message" : {
            "type" : "string"
          },
          "process_id" : {
            "type" : "string"
          },
          "source" : {
            "type" : "string"
          },
          "streams" : {
            "type" : "string",
            "fields" : {
              "keyword" : {
                "type" : "string",
                "index" : "not_analyzed",
                "ignore_above" : 256
              }
            },
            "fielddata" : false
          },
          "timestamp" : {
            "type" : "string"
          }
        }
      }
    }
  }

$ curl -i 'http://localhost:9200/_template?pretty'
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 4

{ }

that means there is no template ?

thank you

9

Yes, correct. That’s an error and if Graylog is unable to create the index template, it should print a warning or error message in its logs.

10

yeah you’re right. please how can I resolve this ? thank you

2018-05-28T12:34:54.882Z ERROR [Indices] Unable to create the Graylog index template: graylog-internal
org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping [message]: analyzer [analyzer_keyword] not found for field [source]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:291) ~[graylog.jar:?]
at org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService.validateAndAddTemplate(MetaDataIndexTemplateService.java:213) ~[graylog.jar:?]
at org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService.access$200(MetaDataIndexTemplateService.java:57) ~[graylog.jar:?]
at org.elasticsearch.cluster.metadata.MetaDataIndexTemplateService$2.execute(MetaDataIndexTemplateService.java:157) ~[graylog.jar:?]
at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[graylog.jar:?]
at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:480) ~[graylog.jar:?]
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:784) ~[graylog.jar:?]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231) ~[graylog.jar:?]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194) ~[graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: org.elasticsearch.index.mapper.MapperParsingException: analyzer [analyzer_keyword] not found for field [source]
at org.elasticsearch.index.mapper.core.TypeParsers.parseAnalyzersAndTermVectors(TypeParsers.java:213) ~[graylog.jar:?]
at org.elasticsearch.index.mapper.core.TypeParsers.parseTextField(TypeParsers.java:250) ~[graylog.jar:?]
at org.elasticsearch.index.mapper.core.StringFieldMapper$TypeParser.parse(StringFieldMapper.java:170) ~[graylog.jar:?]
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:309) ~[graylog.jar:?]
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:222) ~[graylog.jar:?]
at org.elasticsearch.index.mapper.object.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:139) ~[graylog.jar:?]
at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:118) ~[graylog.jar:?]
at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:99) ~[graylog.jar:?]
at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:549) ~[graylog.jar:?]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:319) ~[graylog.jar:?]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:289) ~[graylog.jar:?]
… 11 more

12

Please try restarting Graylog and Elasticsearch and then post the complete logs of your Graylog and Elasticsearch nodes.
:arrow_right: http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

13

1- elasticsearch:
graylog-deprecation.log

[2018-05-31T17:15:30,382][WARN ][o.e.d.e.NodeEnvironment ] ES has detected the [path.data] folder using the cluster name as a folder [/var/lib/elasticsearch], Elasticsearch 6.0 will not allow the cluster name as a folder within the data path
[2018-06-04T08:47:29,480][WARN ][o.e.d.e.NodeEnvironment ] ES has detected the [path.data] folder using the cluster name as a folder [/var/lib/elasticsearch], Elasticsearch 6.0 will not allow the cluster name as a folder within the data path
[2018-06-04T10:09:08,157][WARN ][o.e.d.e.NodeEnvironment ] ES has detected the [path.data] folder using the cluster name as a folder [/var/lib/elasticsearch], Elasticsearch 6.0 will not allow the cluster name as a folder within the data path
[2018-06-04T10:45:42,925][WARN ][o.e.d.e.NodeEnvironment ] ES has detected the [path.data] folder using the cluster name as a folder [/var/lib/elasticsearch], Elasticsearch 6.0 will not allow the cluster name as a folder within the data path

I can’t paste others. so much lines not supported here.

14

Post them to a pastebin service such as https://0bin.net/ or https://gist.github.com/ and share the link.

15

1- elasticsearch-continue

  • graylog.log

https://0bin.net/paste/M2LUr69TZJTHJT8o#pVz2fahlXYpEasFcCtH8zuoM4PKjV8a-uuJLP5rTmVh

2- graylog-server

-server.log

https://0bin.net/paste/LEfjLoJWZFGJT71T#hW7mL3ZBHGoWyYwbeBG75PLU0ePB63PArj+iDHvI2i4

16

Unfortunately both links are incomplete.

17

Sorry. just I replaced it.

18

These are both logs of Graylog nodes, none of Elasticsearch.

Additionally, Graylog is unable to connect to your Elasticsearch node(s):

2018-06-04T14:41:02.581+01:00 ERROR [Messages] Caught exception during bulk indexing: io.searchbox.client.config.exception.CouldNotConnectException: Could not connect to http://127.0.0.1:9200, retrying (attempt #15).
19

web-graylog
web-graylog.PNG1896×845 41.9 KB

Maybee it were when I was making tests.

curl http://127.0.0.1:9200
{
“name” : “UaOkFFS”,
“cluster_name” : “graylog”,
“cluster_uuid” : “H6Z144dsQFK6_sz2kbIrug”,
“version” : {
“number” : “5.6.9”,
“build_hash” : “877a590”,
“build_date” : “2018-04-12T16:25:14.838Z”,
“build_snapshot” : false,
“lucene_version” : “6.6.1”

Do you think it’s because of elasticsearch version : 6.6.1 ?

20

that’s theproblem:

histogram
histogram.PNG1920×969 57.6 KB

21

Unless you’ve edited the output, it’s Elasticsearch 5.6.9 (with Lucene 6.6.1).

What’s in the logs of the Graylog and the Elasticsearch nodes at that time?

22

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.