Indexer failures with netflow

(andrea) #1

Hi at all, i use graylog with mikrotik to monitor network traffic. I have configured a netflow and for 2 weeks it has worked fine. From last week i have a lot of index failure like this

    |an hour ago|graylog_1|396cc270-0b23-11e8-8761-00155d039716|{"type":"mapper_parsing_exception","reason":"failed to parse [date_time]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: \"2018-02-06 10:50:53\" is malformed at \" 10:50:53\""}}|
|an hour ago|graylog_1|38d27e40-0b23-11e8-8761-00155d039716|{"type":"mapper_parsing_exception","reason":"failed to parse [date_time]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: \"2018-02-06 10:50:53\" is malformed at \" 10:50:53\""}}|
|an hour ago|graylog_1|37f2a590-0b23-11e8-8761-00155d039716|{"type":"mapper_parsing_exception","reason":"failed to parse [date_time]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: \"2018-02-06 10:50:51\" is malformed at \" 10:50:51\""}}|
|an hour ago|graylog_1|34d9ccd0-0b23-11e8-8761-00155d039716|{"type":"mapper_parsing_exception","reason":"failed to parse [date_time]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: \"2018-02-06 10:50:45\" is malformed at \" 10:50:45\""}}|

the index failures are maybe 5000 every 24 hours. What’s the problem? i don’t know if is related to the index
changed (from graylog_0 to graylog_1)

thank you


(Jochen) #2

You have to create a custom index mapping for the “date_time” field and make sure it has always the same data type.

(andrea) #3

Hi, but this is the default index and as i said the index has worked fine until last week.

I have created other index with gelf input from exchange server.

(Jochen) #4

And now it doesn’t because the dynamic mapping in Elasticsearch recognized the field as a different type. Maybe it was a string before, now it’s a date.

That’s why you have to create a custom index mapping specifying the data type of that (and any other important) field.

(andrea) #5

ok, but i don’t install new firmware on mikrotik and graylog version is the same version. Why before has worked and now no?

(Jochen) #6

(andrea) #7

Hi, i think to have found the problem, i forget to check this option on new stream that i have created


Now i have one other question about index failure. I create an index for exchgne smtp messages that i send to graylog with nxlog.

sometimes i found on index failure this

{"type":"mapper_parsing_exception","reason":"failed to parse [internal_message_id]","caused_by":{"type":"number_format_exception","reason":"For input string: \"<98059c31840d4bf18a9857097c23efca@

on nxlog configuration the “internal_message_id” is mapped as string field. Why elastic try to convert it in a number?


(Jochen) #8

Same answer as before: Dynamic Mapping.

Create a custom index mapping if you want to prevent that.

(andrea) #9

ok i will try

thank you

(system) #10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.