Indexer failures for timestamp

Graylog Central (peer support)
1

Hi team,

i am using graylog version 2.5 and elasticsearch version 6.5.4, right now i am getting some issue with timestamp

{"type":"mapper_parsing_exception","reason":"failed to parse field [apiTimestamp] of type [date]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: \"12/Jun/2019:07:45:59.055\" is malformed at \"/Jun/2019:07:45:59.055\""}}

can someone please suggest me how to fix this error and how to check which logs creating this error because multiple applications sending logs to my graylog

2

if you share how you ingest data, you might get someone that helps you finding the malfomed log source

3

i am using logstash to send logs to graylog. is this what you are looking for ?
input {
file {
path => “/opt/lsy/log/test/test.log”
type => “testlog”
}
}

filter {
if [type] == “testlog” {
multiline {
negate => true
pattern => “^[%{TIMESTAMP_ISO8601}]”
what => ‘previous’
}
grok {
match => [“message”, “^[%{TIMESTAMP_ISO8601:logdate}]%{SPACE}[%{LOGLEVEL:severity}]%{SPACE}[%{DATA:thread}]%{SPACE}[%{DATA:class}]%{SPACE}%{NUMBER:lineNumber}%{SPACE}|%{SPACE}[%{DATA:componentName}]%{SPACE}|%{SPACE}[%{DATA:componentVersion}]%{SPACE}|%{SPACE}[%{DATA:applicationId}]%{SPACE}|%{SPACE}[%{DATA:clientId}]%{SPACE}|%{SPACE}[%{DATA:customerId}]%{SPACE}|%{SPACE}[%{DATA:busId}]%{SPACE}|%{SPACE}[%{DATA:userId}]%{SPACE}|%{SPACE}[%{DATA:testd}]%{SPACE}|%{SPACE}[%{DATA:apiTimestamp}]%{SPACE}|%{SPACE}%{GREEDYDATA}”]
}
date {
match => [ “logdate”, “ISO8601”]
remove_field => [ “logdate” ]
}
}
if [severity] != “INFO” and [severity] != “WARN” and [severity] != “ERROR” {
drop {}
}
}

output {

gelf {
custom_fields => [‘environment’, ‘{ENVIRONMENT}'] custom_fields => ['service', 'StatusMonitoringService'] custom_fields => ['hostName', '{DOCKER_HOST}’]
custom_fields => [‘containerName’, ‘${CONTAINER_NAME}’]
host => “my host”
port => “my port”
full_message => “”
ignore_metadata => ["@timestamp", “@version”, “severity”, “host”, “source_host”, “source_path”, “short_message”]
}
}

4

the grok for %{DATA:apiTimestamp} will take everything. So if you want to have a timestamp at this, use the pattern for this timestamp that matches. Or create a custom mapping in Elasticsearch for the field apiTimestamp that is of the type string that you are able to ingest everything.

5

Few questions
if I use pattern then it change the date or it will match the pattern and of do not match then ignore it

  • Is custom mapping making date as string will impect searches ??

  • Can we use pipelines to change the date ??

6
  • if a pattern does not match, it will be ignored
  • date, string and number behave different on search, the documentation covers that
  • yes processing pipelines could do all you do in logstash…
7

i have multiple application which is feeding logs in graylog, so is it possible we can check which application or container sending wrong date format ?

second question: if i got mapper_parsing_exception for date will that log ignored by the graylog because i can not see specific timestamp log in graylog

8

Are you talking about the field timestamp or your initial mentioned apiTimestamp ?

Make a custom mapping for apiTimestamp that makes this field a string and look at the field, finding the source with the malformed.

Messages are lost when they could not be inserted to Elasticsearch.

9

after making a custom mapping i can see the culprit services but i am wondering in logstash configuration apiTimestamp is defined [%{DATA:apiTimestamp}] then why elasticsearch treating him as a date i think it should treat as string

10

You think that the data type is decided on extract - but the dataype is decided on field creation in Elasticsearch.

If the first ingested value is a number, the field will be a number. If it is a date, the field will be a date. Only having a custom mapping is forcing a specific type. It does not matter what and how you extract - it does matter what you ingest.

Also after index rotation the game starts over again!

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.