Wrong format in ElasticSearch date mapping

Versions Used: Graylog 4.1.2+20cd592 + ElasticSearch 6.8.10

I have recently started using Graylog and ran into the following phenomenon. I observe the current behaviour:

  1. When looking at the gl-events-template within ElasticSearch’s index templates, I notice that all dates are specified in the following way:
{
  "type": "date",
  "format": "yyyy-MM-dd HH:mm:ss.SSS"
},
  1. When I rotate the active index of the Graylog Events index, which uses the above template, the following mapping is created for all date fields in the new index in ElasticSearch:
{
  "type": "date",
  "format": "8yyyy-MM-dd HH:mm:ss.SSS"
},

Note that the format is invalid, as the years are prepended by an 8.

  1. Now, when looking at the index templates ElasticSearch uses, the original formatting is overwritten. Now all date fields have the wrong formatting as wel:
{
  "type": "date",
  "format": "8yyyy-MM-dd HH:mm:ss.SSS"
},

Does anyone know what the source of this problem might be? Without this being solved, my Graylog instance is close to useless. Thanks in advance!

PS: If more information or configuration is needed, I would be happy to supply that.

@MaxVanDeursen

Might want to see this.

To use java time in 6.8, prefix the date format with an 8. For example, you can change the date format YYYY-MM-dd to 8yyyy-MM-dd to indicate the date format uses java time.

Elasticsearch treats date formats starting with the 8 prefix differently depending on the version:
Java time migration guide | Elasticsearch Guide [7.x] | Elastic

Hi Greg, thanks for your response. That would indeed explain the formatting. The reason I asked was an error in the logs I received, which I wrongly assumed was because of the date formatting:

graylog          | 2021-08-11 08:07:09,335 ERROR: org.graylog.storage.elasticsearch6.IndexFieldTypePollerAdapterES6 - Invalid mapping response: {"simpl_anomalies_8":{"mappings":{"_doc":{"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"application_server":{"type":"keyword"},"class":{"type":"keyword"},"classpath":{"type":"keyword"},"full_message":{"type":"text","analyzer":"standard"},"gl2_accounted_message_size":{"type":"long"},"gl2_processing_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"message":{"type":"text","analyzer":"standard"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"stackTrace":{"type":"text","analyzer":"standard"},"streams":{"type":"keyword"},"timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"}}}}}}
graylog          | 2021-08-11 08:07:09,370 ERROR: org.graylog.storage.elasticsearch6.IndexFieldTypePollerAdapterES6 - Invalid mapping response: {"simpl_workflow_8":{"mappings":{"_doc":{"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"application_server":{"type":"keyword"},"beginState":{"type":"keyword"},"class":{"type":"keyword"},"classpath":{"type":"keyword"},"duration":{"type":"long"},"endState":{"type":"keyword"},"engineService":{"type":"keyword"},"full_message":{"type":"text","analyzer":"standard"},"gl2_accounted_message_size":{"type":"long"},"gl2_processing_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"message":{"type":"text","analyzer":"standard"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"stackTrace":{"type":"text","analyzer":"standard"},"stateTask":{"type":"keyword"},"streams":{"type":"keyword"},"tasksFailed":{"type":"integer"},"tasksNotExecuted":{"type":"integer"},"tasksProcessedInTotal":{"type":"integer"},"tasksSucceeded":{"type":"integer"},"timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"workflow":{"type":"keyword"}}}}}}
graylog          | 2021-08-11 08:07:09,403 ERROR: org.graylog.storage.elasticsearch6.IndexFieldTypePollerAdapterES6 - Invalid mapping response: {"simpl_test_0":{"mappings":{"_doc":{"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"application_server":{"type":"keyword"},"class":{"type":"keyword"},"classpath":{"type":"keyword"},"full_message":{"type":"text","analyzer":"standard"},"gl2_accounted_message_size":{"type":"long"},"gl2_processing_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"message":{"type":"text","analyzer":"standard"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"stackTrace":{"type":"text","analyzer":"standard"},"streams":{"type":"keyword"},"timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"}}}}}}
graylog          | 2021-08-11 08:07:09,441 ERROR: org.graylog.storage.elasticsearch6.IndexFieldTypePollerAdapterES6 - Invalid mapping response: {"simpl_flexmon_demo_7":{"mappings":{"_doc":{"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"application_server":{"type":"keyword"},"class":{"type":"keyword"},"classpath":{"type":"keyword"},"full_message":{"type":"text","analyzer":"standard"},"gl2_accounted_message_size":{"type":"long"},"gl2_processing_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"message":{"type":"text","analyzer":"standard"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"stackTrace":{"type":"text","analyzer":"standard"},"streams":{"type":"keyword"},"timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"}}}}}}
graylog          | 2021-08-11 08:07:14,284 ERROR: org.graylog.storage.elasticsearch6.IndexFieldTypePollerAdapterES6 - Invalid mapping response: {"simpl_api_9":{"mappings":{"_doc":{"dynamic_templates":[{"internal_fields":{"match":"gl2_*","match_mapping_type":"string","mapping":{"type":"keyword"}}},{"store_generic":{"match_mapping_type":"string","mapping":{"type":"keyword"}}}],"properties":{"application_server":{"type":"keyword"},"class":{"type":"keyword"},"classpath":{"type":"keyword"},"full_message":{"type":"text","analyzer":"standard"},"gl2_accounted_message_size":{"type":"long"},"gl2_processing_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"gl2_receive_timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"},"message":{"type":"text","analyzer":"standard"},"request_method":{"type":"keyword"},"request_uri":{"type":"keyword"},"source":{"type":"text","analyzer":"analyzer_keyword","fielddata":true},"stackTrace":{"type":"text","analyzer":"standard"},"status_code":{"type":"short"},"streams":{"type":"keyword"},"timestamp":{"type":"date","format":"8yyyy-MM-dd HH:mm:ss.SSS"}}}}}}

Do you happen to know what would cause these errors? By the way, if posting a new thread is preferred, feel free to remind me!

Hello,

Not 100% sure but the index your using seams that there is something wrong with your index mapping. You can try to rotate your index manually see if that clears it up after you fix your Dat/time issue.

Are you using a custom Index template?

Thanks for your response once again, you were exactly right. I am using a custom index template for all aforementioned indexes, which was the source of the problem. For future reference, I created my index template according to the template supplied in https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-templates.html, e.g.:

{
  "index_patterns": [
    "api*"
  ],
  "order": 1,
  "mappings": {
    "_doc": {
      "properties": {
        "request_uri": {
          "type": "keyword"
        },
        "request_method": {
          "type": "keyword"
        },
        "status_code": {
          "type": "short"
        }
      }
    }
  }
}

Conversely, the Graylog documentation specifies a slightly different template format (https://docs.graylog.org/en/4.0/pages/configuration/elasticsearch.html#creating-a-new-index-template):

{
  "index_patterns": [
    "api*"
  ],
  "order": 1,
  "mappings": {
    "message": {
      "properties": {
        "request_uri": {
          "type": "keyword"
        },
        "request_method": {
          "type": "keyword"
        },
        "status_code": {
          "type": "short"
        }
      }
    }
  }
}

Note that the difference lies in the child element of mapping, which is _doc in the ElasticSearch documentation template, and message in the Graylog documentation template. This was what caused and subsequently fixed the exceptions; simply changing _doc to message in all relevant templates got rid of the exceptions.

2 Likes

@MaxVanDeursen

Awesome, Glad you solved your issue. And thank you for sharing :slight_smile:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.