Custom mapping not applied to indices

[LibraryIT]

Thanks for your guidance - I’m likely just missing something here.

Running:
Graylog v3.1.2+9e96b08
Elasticsearch 6.8.3
Both on same server

Issue:

I need to split a numeric value from log messages, and store it in its own field, then perform operations on those fields as a number (not a string). I am successful in splitting the number from messages to a new field using regex or grok patterns. However, the newly-created field is always stored as a “keyword” type in Elasticsearch. Since the numbers are treated as a keyword, I can’t graph or generate stats for them.

Solutions tried:

Using grok patterns as the extractor type, and setting the datatype to ‘long.’ When using this approach, the new field is still created as a “keyword” datatype in Elasticsearch. No numeric operations are available in Graylog for data in this field.
bytes:%{BASE10NUM:bytes;long}
https://docs.graylog.org/en/3.1/pages/extractors.html#using-grok-patterns-to-extract-data

Creating a custom mapping, adding it, and rotating the active write index. I follow the steps in the Graylog documentation for this topic to the letter, but my custom mapping never appears in the main template, before or after rotating the write index.

Here are my steps. Am I overlooking something?

0. Remove the grok pattern extractor creating the new field that is of the wrong datatype. Rotate the index, so that field is no longer present in the current index. (Goal: avoid index datatype conflicts on that field.)

1. Create a file graylog-custom-mapping.json in /etc/graylog/server:

    {
      "mappings" : {
        "message" : {
          "properties" : {
            "bytes" : {
              "type" : "long"
            },
            "http_response_code" : {
              "type" : "long"
            }
          }
        }
      }
    }
   

2. Add that custom template to Elasticsearch:
   curl -X PUT -d @'graylog-custom-mapping.json' 'http://localhost:9200/_template/graylog-custom-mapping?pretty' -H 'Content-Type: application/JSON'

3. Check the properties mapped in the current Elasticsearch index:
   curl -X GET 'http://localhost:9200/graylog_deflector/_mapping?pretty'
   Of course, the new custom mapping is not included, because the current write index needs to be rotated to load the new template.

4. On the Graylog dashboard, select the Default index set, and rotate the active write index.
   Check again whether the properties for the current index contain the custom mapping. The custom mapping is not present.

I followed the Graylog documentation http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#custom-index-mappings closely. I also consulted Elasticsearch documentation, read past Graylog community topics with a similar question, and lots of Google content.

[lacarrillo]

Hi there,
If you already extracted the field that you need but the field is of type string and you want to convert it to a numeric value what you can do is go to one of the messages that has the field extracted and click create an extractor for that field. While creating the extractor select the option “Copy Input”. You will be redirected to a new page and then you will choose a field name for the new value, add a title and in the option “Add converter” select the Numeric converter and create the extractor. That should do it

[LibraryIT]

Hi lacarrillo,

Thanks for your suggestion. I did what you recommended, but the new field created by the extractor > converter combo is still of “keyword” datatype, and no numeric operations (sum, statistics, graphing) can be performed on the stored values. Any ideas why that might be?

[lacarrillo]

While creating the new field, did you use for the field name the name of a field that existed before ? If you did, try assigning a unique name to the new field, a name that you have never use before to test if it works.

[LibraryIT]

Thanks for that! I just did so - but it did not achieve the desired outcome. A new field, with no previous instances in any index, using a numeric converter, creates a field set to “keyword” type in the active Elasticsearch index.