"value source config is invalid"

GTownson · June 14, 2018, 9:44am

Hi All,

My instance of Graylog stops processing messages for a minute or so, the web interface hangs as well and I see the below error in my Elasticsearch logs. This issue has caused our Graylog instances to go down in some cases.
I have tried to look into the issue and haven’t really got anywhere, has anyone seen this before or knows a solution?

[2018-06-14T09:25:07,592][DEBUG][o.e.a.s.TransportSearchAction] [VJsqk_b] [graylog_4][0], node[VJsqk_bvSIyzXmfWCw-2yg], [P], s[STARTED], a[id=HYfoE-aRSP2oGQFk3aWZ4g]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[graylog_1, graylog_0, graylog_5, graylog_4, graylog_23, graylog_3], indicesOptions=IndicesOptions[id=39, ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[message], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=5, batchedReduceSize=512, preFilterShardSize=64, source={
  "from" : 0,
  "query" : {
    "bool" : {
      "must" : [
        {
          "match_all" : {
            "boost" : 1.0
          }
        }
      ],
      "filter" : [
        {
          "bool" : {
            "must" : [
              {
                "range" : {
                  "timestamp" : {
                    "from" : "2018-06-14 08:24:07.583",
                    "to" : "2018-06-14 08:25:07.583",
                    "include_lower" : true,
                    "include_upper" : true,
                    "boost" : 1.0
                  }
                }
              },
              {
                "query_string" : {
                  "query" : "streams:5b2116c3a8b31a3b1487d8e2",
                  "fields" : [ ],
                  "use_dis_max" : true,
                  "tie_breaker" : 0.0,
                  "default_operator" : "or",
                  "auto_generate_phrase_queries" : false,
                  "max_determinized_states" : 10000,
                  "enable_position_increments" : true,
                  "fuzziness" : "AUTO",
                  "fuzzy_prefix_length" : 0,
                  "fuzzy_max_expansions" : 50,
                  "phrase_slop" : 0,
                  "escape" : false,
                  "split_on_whitespace" : true,
                  "boost" : 1.0
                }
              }
            ],
            "disable_coord" : false,
            "adjust_pure_negative" : true,
            "boost" : 1.0
          }
        }
      ],
      "disable_coord" : false,
      "adjust_pure_negative" : true,
      "boost" : 1.0
    }
  },
  "aggregations" : {
    "gl2_filter" : {
      "filter" : {
        "match_all" : {
          "boost" : 1.0
        }
      },
      "aggregations" : {
        "gl2_terms" : {
          "terms" : {
            "size" : 100,
            "min_doc_count" : 1,
            "shard_min_doc_count" : 0,
            "show_term_doc_count_error" : false,
            "order" : [
              {
                "_count" : "desc"
              },
              {
                "_term" : "asc"
              }
            ]
          }
        }
      }
 ]
          }
        }
      }
    },
    "missing" : {
      "missing" : { }
    }
  }
}}]
org.elasticsearch.transport.RemoteTransportException: [VJsqk_b][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalStateException: value source config is invalid; must have either a field context or a script or marked as unwrapped
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.toValuesSource(ValuesSourceConfig.java:227) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:51) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:225) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactories.createSubAggregators(AggregatorFactories.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorBase.<init>(AggregatorBase.java:78) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.<init>(BucketsAggregator.java:48) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.bucket.SingleBucketAggregator.<init>(SingleBucketAggregator.java:38) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.bucket.filter.FilterAggregator.<init>(FilterAggregator.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.bucket.filter.FilterAggregatorFactory.createInternal(FilterAggregatorFactory.java:72) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:225) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:226) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:55) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:252) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:267) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.9.jar:5.6.9]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
[2018-06-14T09:25:07,592][DEBUG][o.e.a.s.TransportSearchAction] [VJsqk_b] All shards failed for phase: [query]
org.elasticsearch.ElasticsearchException$1: value source config is invalid; must have either a field context or a script or marked as unwrapped
        at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:618) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:126) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:618) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:126) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:241) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:107) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:49) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:217) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:171) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:49) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:217) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1077) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1181) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1159) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:665) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:659) [elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-5.6.9.jar:5.6.9]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.IllegalStateException: value source config is invalid; must have either a field context or a script or marked as unwrapped
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.toValuesSource(ValuesSourceConfig.java:227) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:51) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:225) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactories.createSubAggregators(AggregatorFactories.java:210) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorBase.<init>(AggregatorBase.java:78) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.bucket.BucketsAggregator.<init>(BucketsAggregator.java:48) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.bucket.SingleBucketAggregator.<init>(SingleBucketAggregator.java:38) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.bucket.filter.FilterAggregator.<init>(FilterAggregator.java:52) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.bucket.filter.FilterAggregatorFactory.createInternal(FilterAggregatorFactory.java:72) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:225) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:226) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:55) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:252) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:267) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:674) ~[elasticsearch-5.6.9.jar:5.6.9]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.9.jar:5.6.9]
        ... 3 more

Regards,

George

jochen · June 14, 2018, 9:58am

Please create a bug report at https://github.com/Graylog2/graylog2-server/issues.

GTownson · June 14, 2018, 10:42am

@jochen Is this not an immediately fixable issue? This Graylog instance is quite critical to us, would downgrading resolve the issue?

jochen · June 14, 2018, 10:45am

I don’t know. Someone has to verify the issue. And that’s only going to happen with a bug report.

GTownson · June 18, 2018, 1:55pm

Hi @jochen,

I have created a post on the Elasticsearch forums regarding this issue and they seem to have identified the cause of the issue as the aggregation on the field gl2_terms is missing something.

Here is the link to the forum post, maybe it will shed some more light on the issue.

Cheers,

George

jochen · June 18, 2018, 2:02pm

Please add that to the bug report on GitHub.

GTownson · June 18, 2018, 2:14pm

Where would I find the source code for alerting in the Graylog Github?

Cheers,

George

jochen · June 18, 2018, 2:16pm

Here: GitHub - Graylog2/graylog2-server: Free and open log management

jochen · June 18, 2018, 2:21pm

For reference (linking isn’t that hard):
https://github.com/Graylog2/graylog2-server/issues/4848

GTownson · June 22, 2018, 11:09am

What do you mean “Linking isn’t that hard”?

system · July 6, 2018, 11:09am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Graylog could not load field information: 500 Graylog Central (peer support)	3	1848	May 24, 2019
New to Graylog and receiving error messages Graylog Central (peer support)	9	5586	March 2, 2017
Could not execute search - Graylog Central (peer support)	2	1375	June 12, 2018
Graylog couldn't execute search Graylog Central (peer support)	19	2773	September 6, 2017
GrayLog Crashing Graylog Central (peer support)	9	2098	November 22, 2018

"value source config is invalid"

Related Topics