"Limit of total fields [1000] in index [windows_327] has been exceeded"

Graylog Central (peer support)
1

Hi,

So I have these indexing errors now and then: {“type”:“illegal_argument_exception”,“reason”:“Limit of total fields [1000] in index [windows_327] has been exceeded”}

Which as per the doc is fairly simple to fix and works for some time. First, let me show you I made this json template:

{
"order" : -1,
"template" : "windows_*",
"settings" : {
  "index" : {
    "analysis" : {
      "analyzer" : {
        "analyzer_keyword" : {
          "filter" : "lowercase",
          "tokenizer" : "keyword"
        }
      }
    }
  },
      "index.mapping.total_fields.limit": 5000
},
"mappings" : {
  "message" : {
    "properties" : {
      "message" : {
        "type" : "text",
        "analyzer" : "standard",
        "fielddata" : false
      },
      "full_message" : {
        "type" : "text",
        "analyzer" : "standard",
        "fielddata" : false
      },
      "timestamp" : {
        "type" : "date",
        "format" : "yyyy-MM-dd HH:mm:ss.SSS"
      },
      "source" : {
        "type" : "text",
        "analyzer" : "analyzer_keyword",
        "fielddata" : true
      },
      "streams" : {
        "type" : "keyword"
      }
    },
    "dynamic_templates" : [
      {
        "internal_fields" : {
          "match" : "gl2_*",
          "mapping" : {
            "type" : "keyword"
          }
        }
      },
      {
        "store_generic" : {
          "match" : "*",
          "mapping" : {
            "index" : "not_analyzed"
          }
        }
      }
    ],
    "_source" : {
      "enabled" : true
    }
  }
},
"aliases" : { }
}

which I upload using this command:

curl -X PUT -d @'windows-template.json' 'http://localhost:9200/_template/windows-template?pretty'

After cycling this works nicely for some time but about a week or two after the issue occurs again and I have to apply this ix again.

Is there a way to make this more permanent? I’m running 2.4.6 on Debian Jessie

Thanks

2

depends on your elasticsearch version mainly

3

Voilà:

root@graylog:~# curl -XGET 'localhost:9200'
{
  "name" : "nxPJ_3a",
  "cluster_name" : "graylog",
  "cluster_uuid" : "x42ZtaY7Qni1sNahAwGkEg",
  "version" : {
    "number" : "5.6.3",
    "build_hash" : "1a2f265",
    "build_date" : "2017-10-06T20:33:39.012Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}
4

With a little searching… via Google on Elastisearch+field+limit (I was tempted to send you to www.lmgtfy.com)

This might give you some ideas on how to address. In particular it may be changing your input type so it is not breaking out unique fields for each record. Instead you can use extractors/decorators to get at what you want.

Of course it may be something entirely different. :stuck_out_tongue:

5

I came across that post but it does not seem to be my issue.
He has the same error but he has processing issues whereas I can’t seem to persist the change I commit to ES.

Imho, those two aren’t related.

6

But If you reduce the number of indexed fields by correcting (in his case) the input, you wouldn’t need to bump the ES limit…

Here is a link in ES for increasing the limits for the index as well as on index creation…

7

Well, I understand that but the input are Windows events so managing each possibility would become much more of a problem than applying the current workaround. Plus the current parsing of those events work well for us really.

8

If you can’t get the change to persist you may need to adjust the order of your index template to be applied after Graylog’s index template, not entirely sure how merging works since I never needed more than 1 index template…

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.