Elasticsearch mapping changed from keyword to long

[anonimus (Name)]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
Graylog and elastic logs suddenly filled up with bunch of warnings - mapping errors “failed to parse field [spanId] of type [long] in document with id”

2. Describe your environment:

• OS Information: RHEL7

• Package Version: graylog 4.2

• Service logs, configurations, and environment variables:

3. What steps have you already taken to try and solve the problem?

Turned out, that this spanID field mapping got changed from keyword to long somehow.
Elastic mappings were not touched manually.

4. How can the community help?
Is that some kind of weird glitch or what can change field mapping like that. Does not look like input has any extractors configured. What could have caused this?

Rotating active write index set it back to type keyword and all started working again - messages accepted, warning gone from logs.

:/var/log/elasticsearch # curl -X GET http://127.0.0.1:9200/graylog_3729/_mapping?pretty | grep -i spanid -A 2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3603  100  3603    0     0  2067k      0 --:--:-- --:--:-- --:--:-- 3518k
          --
          "spanId" : {
            "type" : "keyword"
          },
:/var/log/elasticsearch # curl -X GET http://127.0.0.1:9200/graylog_3730/_mapping?pretty | grep -i spanid -A 2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3447  100  3447    0     0  1596k      0 --:--:-- --:--:-- --:--:-- 3366k
        
          "spanId" : {
            "type" : "long"
          },
:/var/log/elasticsearch # curl -X GET http://127.0.0.1:9200/graylog_3731/_mapping?pretty | grep -i spanid -A 2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   520  100   520    0     0  36537      0 --:--:-- --:--:-- --:--:-- 37142


:/var/log/elasticsearch # curl -X GET http://127.0.0.1:9200/graylog_3731/_mapping?pretty | grep -i spanid -A 2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3014  100  3014    0     0  1002k      0 --:--:-- --:--:-- --:--:-- 1471k
          "spanId" : {
            "type" : "keyword"
          },

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

[tmacgbay]

Every time the ElasticSearch Index rotates, the default action is to adjust the field type based on the first data that comes into it. Most likely what happened to you. First item in for SpanId was considered to be a keyword.

The way to specifically avoid this is to create a custom mapping where you specify the field type on index creation. I did an older write up here about this happening to me and how I set the custom mapping and corrected the historical data.

Good luck!

[anonimus (Name)]

I suspected this, read somewhere else. Thanks.

[joe.gross]

@tmacgbay is absolutely correct. When the first thing to hit a newly created index is a number, Elasticsearch sets that field to long, regardless of what it was before. Mapping it permanently to keyword will prevent it in the future, but rotating the index and hoping for a keyword is the right answer.