Wrong format in ElasticSearch date mapping

MaxVanDeursen · August 4, 2021, 9:18am

Versions Used: Graylog 4.1.2+20cd592 + ElasticSearch 6.8.10

I have recently started using Graylog and ran into the following phenomenon. I observe the current behaviour:

When looking at the gl-events-template within ElasticSearch’s index templates, I notice that all dates are specified in the following way:

{
  "type": "date",
  "format": "yyyy-MM-dd HH:mm:ss.SSS"
},

When I rotate the active index of the Graylog Events index, which uses the above template, the following mapping is created for all date fields in the new index in ElasticSearch:

{
  "type": "date",
  "format": "8yyyy-MM-dd HH:mm:ss.SSS"
},

Note that the format is invalid, as the years are prepended by an 8.

Now, when looking at the index templates ElasticSearch uses, the original formatting is overwritten. Now all date fields have the wrong formatting as wel:

{
  "type": "date",
  "format": "8yyyy-MM-dd HH:mm:ss.SSS"
},

Does anyone know what the source of this problem might be? Without this being solved, my Graylog instance is close to useless. Thanks in advance!

PS: If more information or configuration is needed, I would be happy to supply that.

gsmith · August 6, 2021, 12:43am

@MaxVanDeursen

Might want to see this.

To use java time in 6.8, prefix the date format with an 8. For example, you can change the date format YYYY-MM-dd to 8yyyy-MM-dd to indicate the date format uses java time.

Elasticsearch treats date formats starting with the 8 prefix differently depending on the version:
Java time migration guide | Elasticsearch Guide [7.16] | Elastic

MaxVanDeursen · August 11, 2021, 8:11am

Hi Greg, thanks for your response. That would indeed explain the formatting. The reason I asked was an error in the logs I received, which I wrongly assumed was because of the date formatting:

graylog          | 2021-08-11 08:07:09,335 ERROR: org.graylog.storage.elasticsearch6.IndexFieldTypePollerAdapterES6 - Invalid mapping response: {"simpl_anomalies_8":{"mappings":{"_doc":{"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"application_server":{"type":"keyword"},"class":{"type":"keyword"},"classpath":{"type":"keyword"},"full_message":{"type":"text","analyzer":"standard"},"gl2_accounted_message_size":{"type":"long"},"gl2_processing_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"message":{"type":"text","analyzer":"standard"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"stackTrace":{"type":"text","analyzer":"standard"},"streams":{"type":"keyword"},"timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"}}}}}}
graylog          | 2021-08-11 08:07:09,370 ERROR: org.graylog.storage.elasticsearch6.IndexFieldTypePollerAdapterES6 - Invalid mapping response: {"simpl_workflow_8":{"mappings":{"_doc":{"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"application_server":{"type":"keyword"},"beginState":{"type":"keyword"},"class":{"type":"keyword"},"classpath":{"type":"keyword"},"duration":{"type":"long"},"endState":{"type":"keyword"},"engineService":{"type":"keyword"},"full_message":{"type":"text","analyzer":"standard"},"gl2_accounted_message_size":{"type":"long"},"gl2_processing_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"message":{"type":"text","analyzer":"standard"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"stackTrace":{"type":"text","analyzer":"standard"},"stateTask":{"type":"keyword"},"streams":{"type":"keyword"},"tasksFailed":{"type":"integer"},"tasksNotExecuted":{"type":"integer"},"tasksProcessedInTotal":{"type":"integer"},"tasksSucceeded":{"type":"integer"},"timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"workflow":{"type":"keyword"}}}}}}
graylog          | 2021-08-11 08:07:09,403 ERROR: org.graylog.storage.elasticsearch6.IndexFieldTypePollerAdapterES6 - Invalid mapping response: {"simpl_test_0":{"mappings":{"_doc":{"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"application_server":{"type":"keyword"},"class":{"type":"keyword"},"classpath":{"type":"keyword"},"full_message":{"type":"text","analyzer":"standard"},"gl2_accounted_message_size":{"type":"long"},"gl2_processing_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"message":{"type":"text","analyzer":"standard"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"stackTrace":{"type":"text","analyzer":"standard"},"streams":{"type":"keyword"},"timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"}}}}}}
graylog          | 2021-08-11 08:07:09,441 ERROR: org.graylog.storage.elasticsearch6.IndexFieldTypePollerAdapterES6 - Invalid mapping response: {"simpl_flexmon_demo_7":{"mappings":{"_doc":{"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"application_server":{"type":"keyword"},"class":{"type":"keyword"},"classpath":{"type":"keyword"},"full_message":{"type":"text","analyzer":"standard"},"gl2_accounted_message_size":{"type":"long"},"gl2_processing_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"message":{"type":"text","analyzer":"standard"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"stackTrace":{"type":"text","analyzer":"standard"},"streams":{"type":"keyword"},"timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"}}}}}}
graylog          | 2021-08-11 08:07:14,284 ERROR: org.graylog.storage.elasticsearch6.IndexFieldTypePollerAdapterES6 - Invalid mapping response: {"simpl_api_9":{"mappings":{"_doc":{"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"application_server":{"type":"keyword"},"class":{"type":"keyword"},"classpath":{"type":"keyword"},"full_message":{"type":"text","analyzer":"standard"},"gl2_accounted_message_size":{"type":"long"},"gl2_processing_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"message":{"type":"text","analyzer":"standard"},"request_method":{"type":"keyword"},"request_uri":{"type":"keyword"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"stackTrace":{"type":"text","analyzer":"standard"},"status_code":{"type":"short"},"streams":{"type":"keyword"},"timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"}}}}}}

Do you happen to know what would cause these errors? By the way, if posting a new thread is preferred, feel free to remind me!

gsmith · August 17, 2021, 4:25am

Hello,

Not 100% sure but the index your using seams that there is something wrong with your index mapping. You can try to rotate your index manually see if that clears it up after you fix your Dat/time issue.

Are you using a custom Index template?

MaxVanDeursen · August 18, 2021, 6:58am

Thanks for your response once again, you were exactly right. I am using a custom index template for all aforementioned indexes, which was the source of the problem. For future reference, I created my index template according to the template supplied in https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-templates.html, e.g.:

{
  "index_patterns": [
    "api*"
  ],
  "order": 1,
  "mappings": {
    "_doc": {
      "properties": {
        "request_uri": {
          "type": "keyword"
        },
        "request_method": {
          "type": "keyword"
        },
        "status_code": {
          "type": "short"
        }
      }
    }
  }
}

Conversely, the Graylog documentation specifies a slightly different template format (https://docs.graylog.org/en/4.0/pages/configuration/elasticsearch.html#creating-a-new-index-template):

{
  "index_patterns": [
    "api*"
  ],
  "order": 1,
  "mappings": {
    "message": {
      "properties": {
        "request_uri": {
          "type": "keyword"
        },
        "request_method": {
          "type": "keyword"
        },
        "status_code": {
          "type": "short"
        }
      }
    }
  }
}

Note that the difference lies in the child element of mapping, which is _doc in the ElasticSearch documentation template, and message in the Graylog documentation template. This was what caused and subsequently fixed the exceptions; simply changing _doc to message in all relevant templates got rid of the exceptions.

gsmith · August 18, 2021, 9:29pm

@MaxVanDeursen

Awesome, Glad you solved your issue. And thank you for sharing

system · September 1, 2021, 9:29pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Issues with date type converter format string Graylog Central (peer support)	17	4918	November 12, 2021
Help needed on Date Field Graylog Central (peer support)	2	516	January 12, 2019
ElasticSearch Date Issue Graylog Central (peer support)	2	1223	April 6, 2017
Flexible data converter error Graylog Central (peer support)	2	478	April 26, 2019
Recently migrated to another drive now getting lots of index errors Graylog Central (peer support)	6	431	October 3, 2022

Wrong format in ElasticSearch date mapping

Related topics