Indexer failures with netflow

Graylog Central (peer support)
1

Hi at all, i use graylog with mikrotik to monitor network traffic. I have configured a netflow and for 2 weeks it has worked fine. From last week i have a lot of index failure like this

    |an hour ago|graylog_1|396cc270-0b23-11e8-8761-00155d039716|{"type":"mapper_parsing_exception","reason":"failed to parse [date_time]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: \"2018-02-06 10:50:53\" is malformed at \" 10:50:53\""}}|
|---|---|---|---|
|an hour ago|graylog_1|38d27e40-0b23-11e8-8761-00155d039716|{"type":"mapper_parsing_exception","reason":"failed to parse [date_time]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: \"2018-02-06 10:50:53\" is malformed at \" 10:50:53\""}}|
|an hour ago|graylog_1|37f2a590-0b23-11e8-8761-00155d039716|{"type":"mapper_parsing_exception","reason":"failed to parse [date_time]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: \"2018-02-06 10:50:51\" is malformed at \" 10:50:51\""}}|
|an hour ago|graylog_1|34d9ccd0-0b23-11e8-8761-00155d039716|{"type":"mapper_parsing_exception","reason":"failed to parse [date_time]","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: \"2018-02-06 10:50:45\" is malformed at \" 10:50:45\""}}|

the index failures are maybe 5000 every 24 hours. What’s the problem? i don’t know if is related to the index
changed (from graylog_0 to graylog_1)

thank you

Andrea

2

You have to create a custom index mapping for the “date_time” field and make sure it has always the same data type.

http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#custom-index-mappings

3

Hi, but this is the default index and as i said the index has worked fine until last week.

I have created other index with gelf input from exchange server.

4

And now it doesn’t because the dynamic mapping in Elasticsearch recognized the field as a different type. Maybe it was a string before, now it’s a date.

That’s why you have to create a custom index mapping specifying the data type of that (and any other important) field.

5

ok, but i don’t install new firmware on mikrotik and graylog version is the same version. Why before has worked and now no?

6

https://www.elastic.co/guide/en/elasticsearch/guide/2.x/dynamic-mapping.html

7

Hi, i think to have found the problem, i forget to check this option on new stream that i have created

111

Now i have one other question about index failure. I create an index for exchgne smtp messages that i send to graylog with nxlog.

sometimes i found on index failure this

{"type":"mapper_parsing_exception","reason":"failed to parse [internal_message_id]","caused_by":{"type":"number_format_exception","reason":"For input string: \"<98059c31840d4bf18a9857097c23efca@

on nxlog configuration the “internal_message_id” is mapped as string field. Why elastic try to convert it in a number?

thanks

8

Same answer as before: Dynamic Mapping.

Create a custom index mapping if you want to prevent that.

1 Like
9

ok i will try

thank you

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.