Field Statistics, float numbers and Elastic "keyword" type

[rcapp (Ricardo)]

Hello, everyone!

After writing a Grok to extract values from a MariaDB Slow Query log, I noticed that the Field Statics was not validating one of the fields (mariadb_query_time) as a number. In the Grok they were filtered as “%{BASE10NUM:mariadb_query_time;float}”. Then, I changed it to “%{NUMBER:mariadb_query_time;float}”. Saved the Grok and waited for new messages to come.

When checking how these values were been stored, it was possible to see that it is correctly stored as a number for the new occurrences, for instance:

  "mariadb_slow_clientuser" : "graylog_db",
  **"mariadb_query_time"** : 21.133185,
  "mariadb_qc_hit" : "No",
  "mariadb_schema_name" : "nation",          
  "mariadb_lock_time" : 1.09E-4,

But when checking the mapping mariadb_query_time is shown as keyword:

     "mariadb_query" : {
        "type" : "keyword"
      },
      **"mariadb_query_time" : {**
      **  "type" : "keyword"**
     ** },**
      "mariadb_rows_affected" : {
        "type" : "keyword"
      },

I checked other threads with similar questions, but I’m in doubt if custom-index-mappings is what I need and how to apply it.

So, if anyone can help me I’d appreciate.

Thanks!

[rcapp (Ricardo)]

Updating:

I tried using the custom index mapping feature, but to no avail. Then, after some searches on Elastic’s documentation I tried the following:

$ curl -X PUT “localhost:9200/graylog_0/_mapping/_doc?pretty” -H ‘Content-Type: application/json’ -d’
{
“properties”: {
“mariadb_query_time”: {
“type”: “double”
}
}
}
’
{
“error” : {
“root_cause” : [
{
“type” : “illegal_argument_exception”,
“reason” : “mapper [mariadb_query_time] of different type, current_type [keyword], merged_type [double]”
}
],
“type” : “illegal_argument_exception”,
“reason” : “mapper [mariadb_query_time] of different type, current_type [keyword], merged_type [double]”
},
“status” : 400
}

I thought this would update my current value. I don’t have that many messages and re-indexing won’t be that painful, if that is the case.

Any help is appreciated.

Thanks!

[jan (Jan Doberstein)]

he @rcapp

on first ingest elasticsearch guess the field type - if that type had changed for whatever reason, rotate the active write index and it will again guess.

To force a specific field type you need to have a custom mapping.

What will not work at all is changing that after ingest.

[rcapp (Ricardo)]

Hi, @jan!

Thank you for your support: re-indexing made the trick.

I wish I could re-index the others, but there are only 15 msgs on the old index, nothing that we can’t add latter on an external report.

Cheers!