Heavy Backlog on Data Nodes

Tafsir_Alam · August 15, 2018, 11:23pm

Hi,

I can see a huge backlog on approx all data nodes. Can’t able to find the exact reason why this is happening. Can you guys please help me in this to resolve this. Attaching the server log of one of a node having the backlog.

2018-08-15T23:04:04.095Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;56e96e56-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}&gt;

2018-08-15T23:04:04.096Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;56e4b360-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"response_string\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 109, 101, 109, 34, 58, 50, 48, 48, 48, 49, 55, 56, 44, 34, 109, 101, 109, 46, 102, 114, 101, 101, 34, 58, 49, 52, 57, 50]...', original message: bytes can be at most 32766 in length; got 59555","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 59555"}}&gt;

2018-08-15T23:04:04.096Z ERROR [Messages] Failed to index [2] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.

2018-08-15T23:04:05.225Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;58d3d931-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}&gt;

2018-08-15T23:04:05.225Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.

2018-08-15T23:04:09.946Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;5b3c75e4-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"response_string\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 109, 101, 109, 34, 58, 50, 49, 51, 48, 53, 53, 52, 44, 34, 109, 101, 109, 46, 102, 114, 101, 101, 34, 58, 49, 48, 55, 48]...', original message: bytes can be at most 32766 in length; got 59722","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 59722"}}&gt;

2018-08-15T23:04:09.946Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;5b579de0-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"response_string\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 109, 101, 109, 34, 58, 49, 54, 57, 56, 52, 57, 57, 44, 34, 109, 101, 109, 46, 102, 114, 101, 101, 34, 58, 49, 48, 53, 56]...', original message: bytes can be at most 32766 in length; got 59390","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 59390"}}&gt;

2018-08-15T23:04:09.946Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;5b3bd884-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"response_string\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 109, 101, 109, 34, 58, 50, 49, 51, 48, 53, 53, 52, 44, 34, 109, 101, 109, 46, 102, 114, 101, 101, 34, 58, 49, 48, 55, 48]...', original message: bytes can be at most 32766 in length; got 59722","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 59722"}}&gt;

2018-08-15T23:04:09.946Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;5b588842-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"response_string\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 109, 101, 109, 34, 58, 49, 54, 57, 56, 52, 57, 57, 44, 34, 109, 101, 109, 46, 102, 114, 101, 101, 34, 58, 49, 48, 53, 56]...', original message: bytes can be at most 32766 in length; got 59390","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 59390"}}&gt;

2018-08-15T23:04:09.946Z ERROR [Messages] Failed to index [4] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.

2018-08-15T23:04:11.726Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;5d1e2d14-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}&gt;

2018-08-15T23:04:11.726Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.

2018-08-15T23:04:12.765Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;5e308771-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"response_string\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 109, 101, 109, 34, 58, 50, 48, 48, 48, 49, 55, 56, 44, 34, 109, 101, 109, 46, 102, 114, 101, 101, 34, 58, 49, 52, 57, 50]...', original message: bytes can be at most 32766 in length; got 59555","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 59555"}}&gt;

2018-08-15T23:04:12.765Z ERROR [Messages] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: One or more of the items in the Bulk request failed, check BulkResult.getItems() for more information.

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;6021a915-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}&gt;

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;60226c61-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}&gt;

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;60226c62-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}&gt;

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;60229371-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}&gt;

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;6021a911-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}&gt;

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;6021a913-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}&gt;

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;6022ba83-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}&gt;

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;6022ba81-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}&gt;

2018-08-15T23:04:15.929Z WARN [Messages] Failed to index message: index=&lt;graylog_407&gt; id=&lt;6021d021-a0df-11e8-afaf-12e8444bdb34&gt; error=&lt;{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}&gt;

Graylog version - 2.4.6
Elasticsearch - 5.6

Thanks in advance

jan · August 16, 2018, 8:27am

I guess the main reason is already written in the pasted log

Document contains at least one immense term in field=\"response_string\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is

and

{"type":"mapper_parsing_exception","reason":"failed to parse [response_status]","caused_by":{"type":"number_format_exception","reason":"For input string: \"success\""}}

you would solve that if you create your own elasticsearch mapping. It should contain ‘ignore above’ for the field response_string and a specific, maybe string for the field response_status.

http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#custom-index-mappings

Tafsir_Alam · August 16, 2018, 12:39pm

@amitshar04 Hope this will help.

Thanks @jan

system · August 30, 2018, 12:41pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Getting ERROR Message in graylog-server log file Graylog Central (peer support)	4	3514	October 1, 2018
Graylog Index Failures Graylog Central (peer support)	3	1622	July 29, 2020
Document contains at least one immense term ERROR Graylog Central (peer support) elastic	3	384	January 24, 2024
Failed to index Graylog Central (peer support)	2	602	February 17, 2023
Max bytes length exceeded exception - bytes can be at most 32766 in length Graylog Central (peer support)	2	7938	February 7, 2018

Heavy Backlog on Data Nodes

Related Topics