Failed to index

[Kamsy]

Hello my graylog server is enabled and i received the logs but in my /var/log/graylo-server/serevr.log, i have saw this message and in my web interface. how to resolve this problem please.

2023-02-03T11:57:03.278Z ERROR [MessagesAdapterES7] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:
[0]: index [graylog_0], type [_doc], id [decf05a1-a3b9-11ed-8bb7-005056bf8e05], message [ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]
2023-02-03T11:57:37.272Z ERROR [MessagesAdapterES7] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:
[7]: index [graylog_0], type [_doc], id [f3abc030-a3b9-11ed-8bb7-005056bf8e05], message [ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]
2023-02-03T11:58:02.273Z ERROR [MessagesAdapterES7] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:
[2]: index [graylog_0], type [_doc], id [02933601-a3ba-11ed-8bb7-005056bf8e05], message [ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]
2023-02-03T11:58:38.271Z ERROR [MessagesAdapterES7] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:
[0]: index [graylog_0], type [_doc], id [176fc980-a3ba-11ed-8bb7-005056bf8e05], message [ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]
2023-02-03T11:59:03.271Z ERROR [MessagesAdapterES7] Failed to index [1] messages. Please check the index error log in your web interface for the reason. Error: failure in bulk execution:
[1]: index [graylog_0], type [_doc], id [2656f130-a3ba-11ed-8bb7-005056bf8e05], message [ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]]

in my web interface:

14 days ago	graylog_0	9e7597d2-9bc6-11ed-bd02-005056bf8e05	ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]
14 days ago	graylog_0	9e763411-9bc6-11ed-bd02-005056bf8e05	ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]
14 days ago	graylog_0	9e75bee0-9bc6-11ed-bd02-005056bf8e05	ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]
14 days ago	graylog_0	9e76a941-9bc6-11ed-bd02-005056bf8e05	ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]
14 days ago	graylog_0	9e748663-9bc6-11ed-bd02-005056bf8e05	ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]
14 days ago	graylog_0	9e75bee2-9bc6-11ed-bd02-005056bf8e05	ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]
14 days ago	graylog_0	9dd9a5f1-9bc6-11ed-bd02-005056bf8e05	ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]
14 days ago	graylog_0	9dd930c3-9bc6-11ed-bd02-005056bf8e05	ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]
14 days ago	graylog_0	9e75e5f2-9bc6-11ed-bd02-005056bf8e05	ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]
14 days ago	graylog_0	9dd97ee1-9bc6-11ed-bd02-005056bf8e05	ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]
14 days ago	graylog_0	9e74d481-9bc6-11ed-bd02-005056bf8e05	ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]
14 days ago	graylog_0	9e745f52-9bc6-11ed-bd02-005056bf8e05	ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]
14 days ago	graylog_0	9dd957d3-9bc6-11ed-bd02-005056bf8e05	ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has bee

Thanks!

[tmacgbay]

Each index gets capped at 1000 fields. It’s possible to change the cap in Elasticsearch but more than likely it’s because you are capturing data as field names and the field names are different on every message. So… rather than two messages, one with name: Batman , and the next with name: Robin, you may be capturing 234324hd: Batman, j3j38383: Robin. etc. (assuming <FieldName>:<data>)

If you rotate the index, I think it will clear…until that index hits 1000 fields but you really need to make sure the data coming in is clean…as in the field names aren’t incrementing/rotating.