Increase the limit of total fields permanently

Hey Guys, since the bigger use of Graylog i run into an error. It dosnt effect my work but i want to “remove” the failure.

ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]

I dont want to execute the command for every new Index

PUT test_index/_settings
  "index.mapping.total_fields.limit": 5000

Is there a option to change the fields to dynamic or increase the number of fields permanent.
PS: Its a bigger Production with round about 55GiB Logs per day (100.000.000 Messages)

Greeting and
Thanks for your help friends

To my knowledge there is no way to permanently set this, and for good reason. This has negative performance impact on the elasticsearch/opensearch cluster.

Are you able to separate your logs into different streams/index sets to split ip the fields? Also are you parsing json into fields?

You can also try adding a remove_field pipeline rule to remove any unwanted fields. What is nice about this pipeline function is that it accepts regex patterns, so for example, say I wanted to remove ALL fields that start with winlogbeat_ , i could do something like:

rule "TEST remove_field using regex"
    && to_string($message.beats_type) == "winlogbeat"
1 Like

I’m on my way to create streams and cut these log out of the default stream. I don’t know if it possible to remove fields without edit the plain Log? The laws in Germany are very strict for editing possible evidence…

I don’t know if it possible to remove fields without edit the plain Log?

It is unfortunately not possible. This would be “destructive”.

1 Like

If you are just parsing out fields from a message field (how you would with syslog etc) then removing a field won’t edit the message itself. This is notmally also possible with windows logs from winlogbeat etc as it includes both the seperate fields and the message field. The message field will probably show the “original” and you can keep just the fields you need as data for your needs.


Thanks for your help guys. Unfortunately i can give only one “solution tag” but both messages are worth to become the solution. <3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.