Elasticsearch cannot parse timestamp

Hello Community,

today i decided to rotate my index to check if its working as expected.
its the first rotate and i ran into an error i cant solve myself, even after using community and google for quite a time now.

description: after rotating the index, i cant search with following error:
Error Message:
Unable to perform search query failed to parse date field [2018-04-23 22:00:00.000] with format [strict_date_time]
Details:
failed to parse date field [2018-04-23 22:00:00.000] with format [strict_date_time]

Index-site in graylog tells me everything is fine. No errors in graylog-server aswell. But in elasticsearch i found following:

RemoteTransportException[[Cybelle][127.0.0.1:9300][indices:data/read/search[phase/query]]]; nested: ElasticsearchParseException[failed to parse date field [2018-04-23 22:00:00.000] with format [strict_date_time]]; nested: IllegalArgumentException[Parse failure at index [10] of [2018-04-23 22:00:00.000]];                    
[...]
Caused by: java.lang.IllegalArgumentException: Parse failure at index [10] of [2018-04-23 22:00:00.000]
        at org.elasticsearch.common.joda.DateMathParser.parseDateTime(DateMathParser.java:208)

my mappings:

$ curl -X GET 'http://localhost:9200/graylog_deflector/_mapping?pretty'
{
  "graylog_4" : {
    "mappings" : {
      "message" : {
        "dynamic_templates" : [ {
          "internal_fields" : {
            "mapping" : {
              "index" : "not_analyzed",
              "type" : "string"
            },
            "match" : "gl2_*"
          }
        }, {
          "store_generic" : {
            "mapping" : {
              "index" : "not_analyzed"
            },
            "match" : "*"
          }
        } ],
        "properties" : {
          "Programm" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "VPN_Policy" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "app" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "appName" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "appcat" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "appid" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "application_name" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "c" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "catid" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "dst" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "dstMac" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "dstV6" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "dst_city_name" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "dst_country_code" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "dst_geolocation" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "dst_hostname" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "dst_if" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "dst_ip" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "dst_ip_city_name" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "dst_ip_country_code" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "dst_ip_geolocation" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "dst_port" : {
            "type" : "long"
          },
          "facility" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "firmware" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "full_message" : {
            "type" : "string",
            "analyzer" : "standard"
          },
          "fw" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "fw_action" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "fw_city_name" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "fw_country_code" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "fw_geolocation" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "gl2_remote_ip" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "gl2_remote_port" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "gl2_source_input" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "gl2_source_node" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "host_inventory_updated" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "icmpCode" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "id" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "level" : {
            "type" : "long"
          },
          "m" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "mac_address" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "message" : {
            "type" : "string",
            "analyzer" : "standard"
          },
          "msg" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "n" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "note" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "pri" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "process_id" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "proto" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "proto_service" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "proto_type" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "purge_messages" : {
            "type" : "long"
          },
          "repo_action" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "rule" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "sess" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "sid" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "sn" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "source" : {
            "type" : "string",
            "analyzer" : "analyzer_keyword"
          },
          "src" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "srcMac" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "srcV6" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "src_city_name" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "src_country_code" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "src_geolocation" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "src_hostname" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "src_if" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "src_ip" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "src_ip_city_name" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "src_ip_country_code" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "src_ip_geolocation" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "src_port" : {
            "type" : "long"
          },
          "streams" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "subsystem" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "time" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "timestamp" : {
            "type" : "date",
            "format" : "yyyy-MM-dd HH:mm:ss.SSS"
          },
          "tunnel_status" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "type" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "usr" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "z_push_user" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      }
    }
  }
}

tryied to modify the timestamp to “yyyy-MM-dd’T’HH:mm:ss.SSS” got the error, that this timeformat is not supported.

Didnt find any new indeces under /var/lib/elasticsearch
available only “graylog_0”, im at 4 now (tried to modify the format etc. and rotated)

Versions:
CentOS 7.4.1708
Graylog: graylog-server-2.4.3-1
elastic: elasticsearch-2.4.6-1

Any suggestions? - thanks in advance

coffee_is_life

Good morning,

i rolled back my machine to thursday, edit the extractor that could modify the original timestamp (convert to datetime), rotated the index - Error is gone…

since its still in test phase, no data loss etc.

have a nice day

coffee_is_life

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.