New request for custom index template

[Willy (Enrico becchetti)]

Dear friends,
I have been using graylog 4.0 for a long time but now I have an error that I need to fix.

To log messages from windows systems I use a specific input channel and a stream that
forward all of these messages in a particular index (windows).

Unfortunately, as you can see from the message below, the type value is not enough to record everything and it needs to be increased

windows_0 239e0f53-dc23-11ec-9172-001a4a16018f ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] has been exceeded]]

I see some guides and threads without any improvements:

https://docs.graylog.org/docs/elasticsearch

I need to change the default value, 1000, for “type” value using a custom template
for this index.
Can anybody help me to solve this case ?
Thanks a lot
Bye
Willy

[tmacgbay]

You can raise the limit but there are potential drawbacks to that - here is a good Graylog article on it

Also - Here is a previous post where we talked through finding and reducing the number of fields.

Finally of note - I believe the 1000 fields limit is per index set - if you can split out what you are doing into another index set (or more) that may address the issue.

[Willy (Enrico becchetti)]

I read this guide What to Do When You Have 1000+ Fields? | Graylog and I change the value for the index that are in use at the moment.

[root@graylog enrico]# curl -XGET localhost:9200/windows_0/_settings/?pretty
{
“windows_0” : {
“settings” : {
“index” : {
“mapping” : {
“total_fields” : {
“limit” : “2000”
}
},
“number_of_shards” : “8”,
“provided_name” : “windows_0”,
“creation_date” : “1653398764948”,
“analysis” : {
“analyzer” : {
“analyzer_keyword” : {
“filter” : “lowercase”,
“tokenizer” : “keyword”
}
}
},
“number_of_replicas” : “0”,
“uuid” : “XA-XmGRwRd2u3Uwg3DVGug”,
“version” : {
“created” : “7100299”
}
}
}
}
}

now I’d like to know how I can change the default value for new windows_* index
that will be created when graylog make a rotation. I need to set type to 2000.

Thanks

[Willy (Enrico becchetti)]

anybody can help me ?

[tmacgbay]

I still think you need to look at why you are ending up with over 1000 fields - here is an example post of syslog Input with Fortigate where the input needed to be changed to RAW since the Syslog input was creating new fields for each message because it was parsing improperly. Alternatively you can split incoming to different indexes so that you reduce the number of fields in the index.

[Willy (Enrico becchetti)]

Dear all,
I’m using some index , for example one for cisco one for fortigate and so on. The problem is that only inside window.index and I think that the limit that I need to be set is about 1400. The problem is occurr after rotation because when the new index was spawn it has got a default value, 1000. I set a new value only for windows_* index inside elastisearch.yml config file ? Are there any other possibilities ?
Thanks again

[tmacgbay]

You want something like this:

curl -XPUT -H "Content-Type: application/json" \ '<ElasticServer>:9200/<Index_name>/_settings' \ -d '{"index.mapping.total_fields.limit": 2000 }

NOTE: Pulled from here and here.

[Willy (Enrico becchetti)]

Dear Tmcgbay
if you have more time for this case… in the following you’ll see my index “windows” settings:

Windows-index 2 indices, 63,759,830 documents, 13.9GiB

Windows systems logs
Index prefix: windows
Shards: 8
Replicas: 0
Field type refresh interval:5 seconds

Index rotation strategy: Index Time
Rotation period: P1M (1 month, a month)
Index retention strategy: Delete
Max number of indices: 12

now I’ve got two index and your command works fine , but it can be applied only to the windows_0
and windows_1 index, it doesn’t change the default value for the index windows_2,3,4,… which will be generated when the rotation mechanism deems it appropriate.

I hope I have explained my case well.

Thanks
Willy

[tmacgbay]

OK - what you really want is a custom mapping.

Current settings of current index with:

curl  -GET  -H "Content-Type: application/json"  '<elasticsearch>:9200/<index_name>/_settings?pretty'

Create a file that has a JSON settings for your index that contains the following

• index name is winbloat_* (covers all versions for future index rotations)
• file name is winbloat_custom_index.json
• Jump from 1000 default to 2000 total fields

{
  "template": "winbloat_*",
     "settings" : {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "2000"
          }
        }
      }
   }
}

Apply the custom template with:
curl -X PUT -H 'Content-Type: application/json' -d @'winbloat_custom_index.json' 'http://<ElasticSearch>:9200/_template/winbloat_custom_index?pretty'

Remove it if you don’t like it with:

curl -X DELETE -H 'Content-Type: application/json' 'http://<ElasticSearch>:9200/_template/winbloat_custom_index?pretty'

[Willy (Enrico becchetti)]

@tmacgbay

winbloat_

Are you sure about winbloat name ? However I’ll try , many thanks again
Enrico

[tmacgbay]

Didn’t know what your windows index name was so I took a guess…