Heavy Backlog on Data Nodes


(Tafsir) #1

Hi,

I can see a huge backlog on approx all data nodes. Can’t able to find the exact reason why this is happening. Can you guys please help me in this to resolve this. Attaching the server log of one of a node having the backlog.

2018-08-15T23:04:04.095Z WARN [Messages] Failed to index message: index=<graylog_407> id=<56e96e56-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}>

2018-08-15T23:04:04.096Z WARN [Messages] Failed to index message: index=<graylog_407> id=<56e4b360-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"response_string\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 109, 101, 109, 34, 58, 50, 48, 48, 48, 49, 55, 56, 44, 34, 109, 101, 109, 46, 102, 114, 101, 101, 34, 58, 49, 52, 57, 50]...', original message: bytes can be at most 32766 in length; got 59555","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 59555"}}>

2018-08-15T23:04:04.096Z ERROR [Messages] Failed to index [2] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.

2018-08-15T23:04:05.225Z WARN [Messages] Failed to index message: index=<graylog_407> id=<58d3d931-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}>

2018-08-15T23:04:05.225Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.

2018-08-15T23:04:09.946Z WARN [Messages] Failed to index message: index=<graylog_407> id=<5b3c75e4-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"response_string\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 109, 101, 109, 34, 58, 50, 49, 51, 48, 53, 53, 52, 44, 34, 109, 101, 109, 46, 102, 114, 101, 101, 34, 58, 49, 48, 55, 48]...', original message: bytes can be at most 32766 in length; got 59722","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 59722"}}>

2018-08-15T23:04:09.946Z WARN [Messages] Failed to index message: index=<graylog_407> id=<5b579de0-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"response_string\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 109, 101, 109, 34, 58, 49, 54, 57, 56, 52, 57, 57, 44, 34, 109, 101, 109, 46, 102, 114, 101, 101, 34, 58, 49, 48, 53, 56]...', original message: bytes can be at most 32766 in length; got 59390","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 59390"}}>

2018-08-15T23:04:09.946Z WARN [Messages] Failed to index message: index=<graylog_407> id=<5b3bd884-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"response_string\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 109, 101, 109, 34, 58, 50, 49, 51, 48, 53, 53, 52, 44, 34, 109, 101, 109, 46, 102, 114, 101, 101, 34, 58, 49, 48, 55, 48]...', original message: bytes can be at most 32766 in length; got 59722","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 59722"}}>

2018-08-15T23:04:09.946Z WARN [Messages] Failed to index message: index=<graylog_407> id=<5b588842-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"response_string\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 109, 101, 109, 34, 58, 49, 54, 57, 56, 52, 57, 57, 44, 34, 109, 101, 109, 46, 102, 114, 101, 101, 34, 58, 49, 48, 53, 56]...', original message: bytes can be at most 32766 in length; got 59390","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 59390"}}>

2018-08-15T23:04:09.946Z ERROR [Messages] Failed to index [4] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.

2018-08-15T23:04:11.726Z WARN [Messages] Failed to index message: index=<graylog_407> id=<5d1e2d14-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}>

2018-08-15T23:04:11.726Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.

2018-08-15T23:04:12.765Z WARN [Messages] Failed to index message: index=<graylog_407> id=<5e308771-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"response_string\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 109, 101, 109, 34, 58, 50, 48, 48, 48, 49, 55, 56, 44, 34, 109, 101, 109, 46, 102, 114, 101, 101, 34, 58, 49, 52, 57, 50]...', original message: bytes can be at most 32766 in length; got 59555","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 59555"}}>

2018-08-15T23:04:12.765Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=<graylog_407> id=<6021a915-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}>

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=<graylog_407> id=<60226c61-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}>

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=<graylog_407> id=<60226c62-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}>

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=<graylog_407> id=<60229371-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}>

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=<graylog_407> id=<6021a911-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}>

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=<graylog_407> id=<6021a913-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}>

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=<graylog_407> id=<6022ba83-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}>

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=<graylog_407> id=<6022ba81-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}>

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=<graylog_407> id=<6021d021-a0df-11e8-afaf-12e8444bdb34> error=<{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}>

Graylog version - 2.4.6
Elasticsearch - 5.6


Thanks in advance


(Jan Doberstein) #2

I guess the main reason is already written in the pasted log

Document contains at least one immense term in field=\"response_string\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is

and

{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}

you would solve that if you create your own elasticsearch mapping. It should contain ‘ignore above’ for the field response_string and a specific, maybe string for the field response_status.

http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#custom-index-mappings


(Tafsir) #3

@amitshar04 Hope this will help.

Thanks @jan


(system) #4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.