Loading field graph for 'message' with status 500 graylog


(Jay) #1

I’m trying to add the generate chart or quick values to a dashboard. Once I select Quick Values or Generate Chart I get the following error: loading field graph for ‘message’ with status 500 graylog. errors

I found this post: Error with “message” and “source” fields but one of the solutions was to review the UI analysis disabled for fields, which I don’t apparently have.

I restarted the Elasticsearch daemon, but that didn’t resolve anything.

Can someone point me in a direction to dig down deeper?


(Jochen) #2

What’s in the logs of your Graylog and Elasticsearch nodes?
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html


(Jan Doberstein) #3

What Version of Graylog and Elasticsearch did you use?


(Jorge Maia) #4

I’m having the same problem when I try to generate charts or quick values.

Elasticsearch 5.6.7
Graylog 2.4.3


(Jochen) #5

Maybe you should answer the same questions then…



(Jay) #6

Hi,

Graylog: 2.3.2
Elasticsearch: 5.6.5

Graylog log: https://pastebin.com/Hf66k5PM
Elasticsearch log: https://pastebin.com/rHGMQHTt


(Jochen) #7

If you’ve upgraded Elasticsearch recently to version 5.x, try rotating the write-active index (System/Indices/Index Set/Maintenance in the web interface).


(Jay) #8

I seem to be receiving the same error. Have any other suggestions?


(WikiBit) #9

I have a similar issue:

I’ve just setup a first single node (basic) setup of graylog. For testing purposes.
Followed the instructions from the Graylog 2.4 documentation for the out of the box setup, and created an input, capturing logs from a lab firewall.

(Specs: a VM with 8GB RAM, 100GB disk)

But when I go to “SEARCH” in the menu (which defaults to the last 5 minutes) I get 1 or 2 errors:

The error I get all the time:
Error
Loading field graph for ‘message’ failed with status 500

and the Message Graph does not display, in stead it shows “Field graph could not be loaded, please try again after reloading the page.”

Sometimes I also get the error:
Could not update field graph data
Updating field graph data failed: Error: cannot GET http://x.x.x.x:9000/api/search/universal/relative/fieldhistogram?query=*&range=300&interval=minute&field=message&cardinality=false (500)

Then the Histogram also does not show.

anyone an idea how to fix this?

When I get the first error, I see the below in /var/log/elasticsearch/graylog.log:

[2018-04-01T13:41:53,608][DEBUG][o.e.a.s.TransportSearchAction] [7HqU0Vm] [graylog_0][3], node[7HqU0VmWSMOAmmx7Ynio-Q], [P], s[STARTED], a[id=MJrhS6G9TqqulK_WkgIrDQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[graylog_0], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[message], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=5, batchedReduceSize=512, preFilterShardSize=64, source={
  "from" : 0,
  "query" : {
    "bool" : {
      "must" : [
        {
          "match_all" : {
            "boost" : 1.0
          }
        }
      ],
      "filter" : [
        {
          "bool" : {
            "must" : [
              {
                "range" : {
                  "timestamp" : {
                    "from" : "2018-04-01 11:36:53.600",
                    "to" : "2018-04-01 11:41:53.600",
                    "include_lower" : true,
                    "include_upper" : true,
                    "boost" : 1.0
                  }
                }
              }
            ],
            "disable_coord" : false,
            "adjust_pure_negative" : true,
            "boost" : 1.0
          }
        }
      ],
      "disable_coord" : false,
      "adjust_pure_negative" : true,
      "boost" : 1.0
    }
  },
  "aggregations" : {
    "gl2_histogram" : {
      "date_histogram" : {
        "field" : "timestamp",
        "interval" : "1m",
        "offset" : 0,
        "order" : {
          "_key" : "asc"
        },
        "keyed" : false,
        "min_doc_count" : 0
      },
      "aggregations" : {
        "gl2_stats" : {
          "stats" : {
            "field" : "message"
          }
        }
      }
    }
  }
}}] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [7HqU0Vm][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [message] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
        at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:336) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:111) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:166) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.resolve(ValuesSourceConfig.java:96) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.resolveConfig(ValuesSourceAggregationBuilder.java:297) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:290) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder$LeafOnly.doBuild(ValuesSourceAggregationBuilder.java:42) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:126) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:347) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.aggregations.AggregatorFactory.<init>(AggregatorFactory.java:187) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.<init>(ValuesSourceAggregatorFactory.java:40) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregatorFactory.<init>(DateHistogramAggregatorFactory.java:53) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.innerBuild(DateHistogramAggregationBuilder.java:331) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:291) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:39) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:126) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:347) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:655) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:485) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:461) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.8.jar:5.6.8]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.8.jar:5.6.8]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

(Jay) #10

I haven’t found a solution yet. I’m still trying to figure it out. My system doesn’t allow internet access behind my FW. I’m curious, if I take this VM to another site and put it online, if it would resolve itself.


(WikiBit) #11

My system has standard internet access. So I doubt the issues has to do with that.

I’m not a graylog expert or developer, but can it have something to do with the size or type of messages which are sent to the graylog server?

My installation is fairly vanilla, and at the moment, only getting messages from my Pfsense firewall


(WikiBit) #12

Still looking for a solution.
I’ve been searching on the web, but nothing comes close to a solution


(system) #13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.