Is there a way to change (customize) the index mapping?
The reason i need this is, that we are ingesting firewall-logs and the “number_of_bytes_in / out” data is displayed as a nummeric value, resulting into following error:
OpenSearch exception [type=illegal_argument_exception, reason=Field [number_of_bytes_in] of type [keyword] is not supported for aggregation [sum]].
The error occures when creating a new aggregation with following metrics:
Environment:
Greetings!
Elasticsearch (and OpenSearch) use something called Dynamic mapping. The first message in the index is used determine what type of data is in each field, and then this is set as the field type for that index. This cannot be changed once it is set. However, the index can be rotated and it can “try again”.
To resolve this you have a couple of options:
You can do this with a pipeline rule such as:
rule "force number_of_bytes_in to int"
when
has_field("number_of_bytes_in")
then
set_field("number_of_bytes_in", to_long($message.number_of_bytes_in));
end
This important part above is the to_long() function, which forces the data to be saved as an integer. This should cause dynamic mappings to detect the field as a number without any issue.
This is a more advanced topic, and I definitely recommend you try it out in a lab or test environment first. See Elasticsearch Versions .
In short, you can tell Elasticsearch/OpenSearch explicitly what field type each field should be. This will be used instead of dynamic mappings so the field will always have the correct type.
Once you do get the field typed correctly (e.g. as an integer) you can rotate your index set and the new index will use the new typing. Keep in mind the past indexes will always have the old field type and cannot be shown on an aggregation that require an integer.
Let me know if you have any questions (I know thats a lot!)
Thanks a lot for your answer. I will stick with option 1 at the moment. Do I need to rotate the index here as well?
You will need to rotate the index in order for the new field type to take effect.
Is there a way to force the index rotation?
There is:
Perfect. Thank you very much!
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.