Upgrading to Elasticsearch 7.10 / problem with custom index template

Graylog Central (peer support)
5

@H2Cyber @zoulja

The Graylog documentation prescribes this as a template:

{
  "template": "graylog_*",
  "mappings" : {
    "message" : {
      "properties" : {
        "http_method" : {
          "type" : "keyword"
        },
        "http_response_code" : {
          "type" : "long"
        },
        "ingest_time" : {
          "type" : "date",
          "format": "strict_date_time"
        },
        "took_ms" : {
          "type" : "long"
        }
      }
    }
  }
}

With ES 6.8 this worked fine, we used this model and the ES documentation to build what we needed. However, with ES 7.10, suddenly no messages were being indexed in the set with that custom template applied, even after we rotated the active index. We tried removing and reloading the custom template, but received the following error:

"error" : {
"root_cause" : [
  {
    "type" : "mapper_parsing_exception",
    "reason" : "Root mapping definition has unsupported parameters:  [message : {properties={<redacted our custom properties>}]"
  }
}

}

So, we looked at the pertinent deflector and reviewed the documentation and determined that we just needed to remove the ‘message’ structure and move ‘properties’ and its children a level up.

{
  "template": "graylog_*",
  "mappings" : {
  "properties" : {
	"http_method" : {
	  "type" : "keyword"
	},
	"http_response_code" : {
	  "type" : "long"
	},
	"ingest_time" : {
	  "type" : "date",
	  "format": "strict_date_time"
	},
	"took_ms" : {
	  "type" : "long"
	}
  }
  }
}

Once we did that everything worked as expected. We did lose some data in the process.

3 Likes