Change elasticsearch field mappings cause error in query

Graylog Central (peer support)
1

@jan
We had the same issue with a “numeric” field for the value of httpd_responsetime.
Currently we are not able to make a graph out of it.

We created a new index template, and we are using a different index than the default one, let’s call it “shop” index.

{
  "template": "shop*",
  "mappings" : {
    "message" : {
      "properties" : {
        "httpd_access_time" : {
          "type" : "long"
        }
      }
    }
  }
}

Then we uploaded it into ES as described here:
http://docs.graylog.org/en/2.3/pages/configuration/elasticsearch.html#custom-index-mappings

We now get an error like this in ES:
Caused by: org.elasticsearch.search.aggregations.AggregationExecutionException: Merging/Reducing the aggregations failed when computing the aggregation [gl2_terms] because the field you gave in the aggregation query existed as two different types in two different indices

Do we need to delete the old indices now cintaining the old fields?
The query is limited to 5 minutes, so it should only query the recent index.

thanks!

2

Hej @shakalandy

the main reason for that is quite simple. Your new indices contains the new type, but old ones that might be present contains the old type and the query you do might ask the old indices.

You could rebuild your index stats to ensure Graylog knows what timestamps are in what index. If you still hit other indices you might localize a bug.

In addition I had moved your question into a new topic - feel free to edit the topic.

3

Thanks for the fast reply @jan.

Do you mean to “Recalculate index ranges”? This won’t help.
Or is this another maintenance operation?

4

@shakalandy - yes that “recalculate index ranges” is exactly that.

If you check the index set ( System > indices > Index name ) and look at the message Contains messages up to ... does one of the older indices would fit into the search window?

If you feel that this is bug, please create a bug report over at github.

thanks

5

No, unfortunately its the recent one which should fit.
Tried to create a new write set…

how to query the index to get its field configurations?

got it:

the new index has

  "httpd_access_time" : {
            "type" : "long"
          },

will recheck tomorrow…

6

i think i got it @jan.

The Message is routed to the new stream via pipeline processing, but also exist in the Default Stream.
Is there a function to remove it from the default stream when routing to a custom one?

Solution with 2.4? https://github.com/Graylog2/graylog-plugin-pipeline-processor/issues/221

7

yes @shakalandy

with 2.4 you can a) remove from default while routing to a stream, but in addition you have “remove from stream” where you can remove the default stream

    remove_from_stream(id: "000000000000000000000001");

add after you had routed to the destination stream.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.