# Wrong format in ElasticSearch date mapping

**URL:** <https://community.graylog.org/t/wrong-format-in-elasticsearch-date-mapping/20764>\
**Category:** Graylog Tech Challenges\
**Created:** [August 4, 2021, 9:18am UTC](https://community.graylog.org/t/wrong-format-in-elasticsearch-date-mapping/20764 "2021-08-04T09:18:46Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![MaxVanDeursen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/maxvandeursen/32/9265_2.png) [@MaxVanDeursen](https://community.graylog.org/u/MaxVanDeursen)\
**Post date:** [August 4, 2021, 9:18am UTC](https://community.graylog.org/t/wrong-format-in-elasticsearch-date-mapping/20764/1 "2021-08-04T09:18:46Z")

</div>

**Versions Used** : Graylog 4.1.2+20cd592 + ElasticSearch 6.8.10

I have recently started using Graylog and ran into the following phenomenon. I observe the current behaviour:

1. When looking at the `gl-events-template` within ElasticSearch’s index templates, I notice that all dates are specified in the following way:

```auto
{
  "type": "date",
  "format": "yyyy-MM-dd HH:mm:ss.SSS"
},

```

1. When I rotate the active index of the `Graylog Events` index, which uses the above template, the following mapping is created for all date fields in the new index in ElasticSearch:

```auto
{
  "type": "date",
  "format": "8yyyy-MM-dd HH:mm:ss.SSS"
},

```

Note that the format is invalid, as the years are prepended by an `8`.

1. Now, when looking at the index templates ElasticSearch uses, the original formatting is overwritten. Now all date fields have the wrong formatting as wel:

```auto
{
  "type": "date",
  "format": "8yyyy-MM-dd HH:mm:ss.SSS"
},

```

Does anyone know what the source of this problem might be? Without this being solved, my Graylog instance is close to useless. Thanks in advance!

PS: If more information or configuration is needed, I would be happy to supply that.

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [August 6, 2021, 12:43am UTC](https://community.graylog.org/t/wrong-format-in-elasticsearch-date-mapping/20764/2 "2021-08-06T00:43:44Z")

</div>

@MaxVanDeursen

Might want to see this.

> To use java time in 6.8, prefix the date format with an 8. For example, you can change the date format YYYY-MM-dd to 8yyyy-MM-dd to indicate the date format uses java time.
> 
> Elasticsearch treats date formats starting with the 8 prefix differently depending on the version:  
> [Java time migration guide | Elasticsearch Guide [7.16] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/migrate-to-java-time.html#java-time-convert-date-formats)

---

<div class="post-metadata">

**Author:** ![MaxVanDeursen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/maxvandeursen/32/9265_2.png) [@MaxVanDeursen](https://community.graylog.org/u/MaxVanDeursen)\
**Post date:** [August 11, 2021, 8:11am UTC](https://community.graylog.org/t/wrong-format-in-elasticsearch-date-mapping/20764/3 "2021-08-11T08:11:27Z")

</div>

Hi Greg, thanks for your response. That would indeed explain the formatting. The reason I asked was an error in the logs I received, which I wrongly assumed was because of the date formatting:

```auto
graylog | 2021-08-11 08:07:09,335 ERROR: org.graylog.storage.elasticsearch6.IndexFieldTypePollerAdapterES6 - Invalid mapping response: {"simpl_anomalies_8":{"mappings":{"_doc":{"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"application_server":{"type":"keyword"},"class":{"type":"keyword"},"classpath":{"type":"keyword"},"full_message":{"type":"text","analyzer":"standard"},"gl2_accounted_message_size":{"type":"long"},"gl2_processing_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"message":{"type":"text","analyzer":"standard"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"stackTrace":{"type":"text","analyzer":"standard"},"streams":{"type":"keyword"},"timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"}}}}}}
graylog | 2021-08-11 08:07:09,370 ERROR: org.graylog.storage.elasticsearch6.IndexFieldTypePollerAdapterES6 - Invalid mapping response: {"simpl_workflow_8":{"mappings":{"_doc":{"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"application_server":{"type":"keyword"},"beginState":{"type":"keyword"},"class":{"type":"keyword"},"classpath":{"type":"keyword"},"duration":{"type":"long"},"endState":{"type":"keyword"},"engineService":{"type":"keyword"},"full_message":{"type":"text","analyzer":"standard"},"gl2_accounted_message_size":{"type":"long"},"gl2_processing_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"message":{"type":"text","analyzer":"standard"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"stackTrace":{"type":"text","analyzer":"standard"},"stateTask":{"type":"keyword"},"streams":{"type":"keyword"},"tasksFailed":{"type":"integer"},"tasksNotExecuted":{"type":"integer"},"tasksProcessedInTotal":{"type":"integer"},"tasksSucceeded":{"type":"integer"},"timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"workflow":{"type":"keyword"}}}}}}
graylog | 2021-08-11 08:07:09,403 ERROR: org.graylog.storage.elasticsearch6.IndexFieldTypePollerAdapterES6 - Invalid mapping response: {"simpl_test_0":{"mappings":{"_doc":{"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"application_server":{"type":"keyword"},"class":{"type":"keyword"},"classpath":{"type":"keyword"},"full_message":{"type":"text","analyzer":"standard"},"gl2_accounted_message_size":{"type":"long"},"gl2_processing_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"message":{"type":"text","analyzer":"standard"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"stackTrace":{"type":"text","analyzer":"standard"},"streams":{"type":"keyword"},"timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"}}}}}}
graylog | 2021-08-11 08:07:09,441 ERROR: org.graylog.storage.elasticsearch6.IndexFieldTypePollerAdapterES6 - Invalid mapping response: {"simpl_flexmon_demo_7":{"mappings":{"_doc":{"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"application_server":{"type":"keyword"},"class":{"type":"keyword"},"classpath":{"type":"keyword"},"full_message":{"type":"text","analyzer":"standard"},"gl2_accounted_message_size":{"type":"long"},"gl2_processing_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"message":{"type":"text","analyzer":"standard"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"stackTrace":{"type":"text","analyzer":"standard"},"streams":{"type":"keyword"},"timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"}}}}}}
graylog | 2021-08-11 08:07:14,284 ERROR: org.graylog.storage.elasticsearch6.IndexFieldTypePollerAdapterES6 - Invalid mapping response: {"simpl_api_9":{"mappings":{"_doc":{"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"application_server":{"type":"keyword"},"class":{"type":"keyword"},"classpath":{"type":"keyword"},"full_message":{"type":"text","analyzer":"standard"},"gl2_accounted_message_size":{"type":"long"},"gl2_processing_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"message":{"type":"text","analyzer":"standard"},"request_method":{"type":"keyword"},"request_uri":{"type":"keyword"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"stackTrace":{"type":"text","analyzer":"standard"},"status_code":{"type":"short"},"streams":{"type":"keyword"},"timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"}}}}}}

```

Do you happen to know what would cause these errors? By the way, if posting a new thread is preferred, feel free to remind me!

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [August 17, 2021, 4:25am UTC](https://community.graylog.org/t/wrong-format-in-elasticsearch-date-mapping/20764/4 "2021-08-17T04:25:55Z")

</div>

Hello,

Not 100% sure but the index your using seams that there is something wrong with your index mapping. You can try to rotate your index manually see if that clears it up after you fix your Dat/time issue.

> **[Get Mapping | Elasticsearch Guide \[6.8\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-get-mapping.html)**

Are you using a custom Index template?

---

<div class="post-metadata">

**Author:** ![MaxVanDeursen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/maxvandeursen/32/9265_2.png) [@MaxVanDeursen](https://community.graylog.org/u/MaxVanDeursen)\
**Post date:** [August 18, 2021, 6:58am UTC](https://community.graylog.org/t/wrong-format-in-elasticsearch-date-mapping/20764/5 "2021-08-18T06:58:24Z")

</div>

Thanks for your response once again, you were exactly right. I am using a custom index template for all aforementioned indexes, which was the source of the problem. For future reference, I created my index template according to the template supplied in [https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-templates.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-templates.html), e.g.:

```auto
{
  "index_patterns": [
    "api*"
  ],
  "order": 1,
  "mappings": {
    "_doc": {
      "properties": {
        "request_uri": {
          "type": "keyword"
        },
        "request_method": {
          "type": "keyword"
        },
        "status_code": {
          "type": "short"
        }
      }
    }
  }
}

```

Conversely, the Graylog documentation specifies a slightly different template format ([https://docs.graylog.org/en/4.0/pages/configuration/elasticsearch.html#creating-a-new-index-template](https://docs.graylog.org/en/4.0/pages/configuration/elasticsearch.html#creating-a-new-index-template)):

```auto
{
  "index_patterns": [
    "api*"
  ],
  "order": 1,
  "mappings": {
    "message": {
      "properties": {
        "request_uri": {
          "type": "keyword"
        },
        "request_method": {
          "type": "keyword"
        },
        "status_code": {
          "type": "short"
        }
      }
    }
  }
}

```

Note that the difference lies in the child element of `mapping`, which is `_doc` in the ElasticSearch documentation template, and `message` in the Graylog documentation template. This was what caused and subsequently fixed the exceptions; simply changing `_doc` to `message` in all relevant templates got rid of the exceptions.

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [August 18, 2021, 9:29pm UTC](https://community.graylog.org/t/wrong-format-in-elasticsearch-date-mapping/20764/6 "2021-08-18T21:29:46Z")

</div>

@MaxVanDeursen

Awesome, Glad you solved your issue. And thank you for sharing 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [September 1, 2021, 9:29pm UTC](https://community.graylog.org/t/wrong-format-in-elasticsearch-date-mapping/20764/7 "2021-09-01T21:29:51Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.