Graylog custom mapping for several templates

[ionzion (Ion)]

Description of your problem

Hi folks, may be somebody could advise me how to apply custom mapping for several templates through graylog-custom-mapping-7x.json.

I’ve tried the following ways:
{
“template”: “gc-prod-",
“settings”: {
“index.mapping.total_fields.limit”: 2000
}
},
{
“template”: "dfa_”,
“mappings”: {
“properties”: {
“app_time”: {
“type”: “date”,
“format”: “yyyy-MM-dd’T’HH:mm:ssZZZZZ”
}
}
}
}

or

{
“template”: [“gc-prod-","dfa_”],
“settings”: {
“index.mapping.total_fields.limit”: 2000
},
“mappings”: {
“properties”: {
“app_time”: {
“type”: “date”,
“format”: “yyyy-MM-dd’T’HH:mm:ssZZZZZ”
}
}
}
}

The only working way is to specify the * for index pattern.
{
“template”: “*”,
“settings”: {
“index.mapping.total_fields.limit”: 2000
},
“mappings”: {
“properties”: {
“app_time”: {
“type”: “date”,
“format”: “yyyy-MM-dd’T’HH:mm:ssZZZZZ”
}
}
}
}

Output:
{
“graylog-custom-mapping” : {
“order” : 0,
“index_patterns” : [
“*”
],
“settings” : {
“index” : {
“mapping” : {
“total_fields” : {
“limit” : “2000”
}
}
}
},
“mappings” : {
“properties” : {
“app_time” : {
“format” : “yyyy-MM-dd’T’HH:mm:ssZZZZZ”,
“type” : “date”
}
}
},
“aliases” : { }
}
}

Operating system information

• Debian 10

Package versions

• Graylog 4.0.7
• MongoDB 4.2.14
• Elasticsearch 7.10

[tmacgbay]

@ionzion

“template”: "gc-prod- ",

I can advise, but you aren’t clear on what you actually want… so I am not 100% sure of what you are trying to accomplish… I am guessing that you want to create custom templates for two indexes, one starting with gc-prod and the other starting with dfa_ but it’s not working unless you wildcard the whole index name?

If so, you can use the asterisk "*" to cover the rest of the template name so it would look like this:

"template": "gc-prod-*",
	"settings": {
		"index.mapping.total_fields.limit": 2000
	}
},
{
"template": "dfa_*",
	"mappings": {
		"properties": {
			"app_time": {
				"type": "date",
				"format": "yyyy-MM-dd'T'HH:mm:ssZZZZZ"
			}
		}
	}
}

Notice how I used the forum tools (like: </> ) to make the code look nice and I indented for readability? These things, and being more specific about what you are trying to accomplish make it easier for the person who is trying to help you in their free time…

[ionzion (Ion)]

Hi @tmacgbay, thank you for reply and your advice. Yes you’ve understood right, I want to create custom template for two indexes gc-prod-* and dfa_*. I’ve tried the method you advised earlier and it didn’t work.
If I check what I’ve applied it shows:
curl -X GET ‘http://localhost:9200/_template/graylog-custom-mapping?pretty’

{
  "graylog-custom-mapping" : {
    "order" : 0,
    "index_patterns" : [
      "gc-prod-*"
    ],
    "settings" : {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "2000"
          }
        }
      }
    },
    "mappings" : { },
     "aliases" : { }
  }
}

So it applies only the first template the second template with:

{
"template": "dfa_*",
  "mappings": {
    "properties": {
      "app_time": {
        "type": "date",
        "format": "yyyy-MM-dd'T'HH:mm:ssZZZZZ"
      }
    }
  }
}

is missing. The only way I’ve found to apply this is to put all the mappings and settings in one template for all indexes with index pattern “*”. But this way I think is to dirty, so that is why I ask if there is one other way to do that.

[tmacgbay]

Try putting them in as separate custom mappings, or really separate custom mapping files that you apply. Where the file names are gl_custom_gc-prod.json and gl_custom_dfa.json

curl -X PUT  -H 'Content-Type: application/json' -d @'gl_custom_gc-prod.json' 'http://elstc-main:9200/_template/gl_custom_gc-prod?pretty

curl -X PUT -H 'Content-Type: application/json' -d @'gl_custom_dfa.json' 'http://elstc-main:9200/_template/gl_custom_dfa?pretty

Just be sure to keep track of the customizations you put in. You can always query them but it’s better to document!

[ionzion (Ion)]

thank you @tmacgbay, this worked!