Graylog-internal total field limit

[hebval]

I have some indexing failures. I tried to fix them by increasing the index.mapping.total_fields.limit in my index, but I forgot they had a 4-day rotation, so I have to set the index.mapping.total_fields.limit manually everytime, for each new index. (graylog_277, graylog_278…)

I’d like to know if it’s possible to set the index.mapping.total_fields.limit value by default in my Default Index Set, so all the new index will have the same index.mapping.total_fields.limit value, matching my requirements.

I used this command for each index :

curl -X PUT localhost:9200/graylog_277/_settings -H ‘Content-Type: application/json’ -d’{ “index.mapping.total_fields.limit”: 1100 }’

Is there a similar command to change the Default Index Set value ?

[gsmith]

Hello && Welcome @hebval

@hebval

manually everytime, for each new index.

That is correct, If your using the default ES index template I would check out this documentation here

The command used below, which I think you know now is just for that INDEX set, You could either create a new index template of try to modify the default index template that Elasticsearch uses for Graylog. If you do that insure you rotate the index so the new settings are enbled.

@hebval

curl -X PUT localhost:9200/graylog_277/_settings -H ‘Content-Type: application/json’ -d’{ “index.mapping.total_fields.limit”: 1100 }’

If your stuck on what index template to use a cURL command could help like so.

curl -X GET localhost:9200/_cat/templates

Results

[root@graylog plugin]# curl -X GET localhost:9200/_cat/templates
beats-template                  [beats_*]                    -1
.ml-meta                        [.ml-meta]                   0          6081399
.transform-internal-005         [.transform-internal-005]    0          7100099
.triggered_watches              [.triggered_watches*]        2147483647
.ml-config                      [.ml-config]                 0          6081399
batman-template                 [batman_*]                   -1
restored-archive-template       [restored-archive*]          -1
linuxserver-template            [linuxserver_*]              -1
gl-events-template              [gl-events_*]                -1
gl-failures-template            [gl-failures_*]              -1
.monitoring-beats               [.monitoring-beats-6-*]      0          6070299
.monitoring-alerts              [.monitoring-alerts-6]       0          6070299
.monitoring-es                  [.monitoring-es-6-*]         0          6070299
security-index-template         [.security-*]                1000
gl-system-events-template       [gl-system-events_*]         -1
.transform-notifications-000002 [.transform-notifications-*] 0          7100099
.logstash-management            [.logstash]                  0
.ml-notifications               [.ml-notifications]          0          6081399
.monitoring-logstash            [.monitoring-logstash-6-*]   0          6070299
firewall-template               [firewall_*]                 -1
.monitoring-kibana              [.monitoring-kibana-6-*]     0          6070299
graylog-internal                [graylog_*]                  -1
graylog-custom-mapping          [mytemplog]                  0
netflow-template                [netflow_*]                  -1
.ml-anomalies-                  [.ml-anomalies-*]            0          6081399
.watches                        [.watches*]                  2147483647
.watch-history-9                [.watcher-history-9*]        2147483647
.ml-state                       [.ml-state*]                 0          6081399
[root@graylog plugin]# 

If you noticed this section…

graylog-internal                [graylog_*]                  -1

I’m using that Index prefix.

I haven’t done that yet, hope that helps

[hebval]

Hi @gsmith ! Thanks for replying.

So it is possible to edit the default template, that’s nice.

Now what would be my command to do it ?

curl -X PUT localhost:9200/?/?
-H ‘Content-Type: application/json’ -d’{ “index.mapping.total_fields.limit”: 1100 }’

I don’t know what to put after 9200…

[gsmith]

Hello,

I’m not sure, because I haven’t done that yet so I cant really give you the fix , sorry.
Maybe someone else here has. TBH dig through the forum I’m sure someone has done this before.
Specially with this error:

ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]