How to handle elasticsearch mapping error due to different data type

[umeshdc]

Hi All,

We have a message field called ‘status’ which can have either number(200) or string (‘Pending’).
Currently we are using elasicsearch dynamic mapping and due to this we are getting numberFormat exceptions whenever we have string data to status field.

Example:
When first message comes with status field value 200 to elasticsearch, it maps this field with data type as long. later after if status field get some string data then we will have numberFormat exception.

Tried below solutions,

1. We tried to change the status field mapping on elasticsearch, but unable to convert field type from long to text
2. We have looked around changing default static mapping from graylog2, but this limits us to have fixed set of fields in the log.

Please let us know for the best approach in order to solve this issue.

[jochen]

You could use the Processing Pipelines to enforce a fixed data type for the “status” field and either convert it into the correct type or rename the field.

http://docs.graylog.org/en/2.4/pages/pipelines.html

[jan (Jan Doberstein)]

use some kind of processing pipeline rule like mentioned in this posting:

Howto figure out the cause of a field type conflict

I can think of a processing pipeline that checks the field dst_port not beeing a number and then rename it - or write a debug log entry in the graylog server.log that contains the message source (and then delete the message). Something like the following (this is untested!) rule "dst_port_not_long" when has_field("dst_port") AND to_long($message.dst_port) == 0 then rename_field("dst_port", "dst_port_nn") end

to have one status field with number and one with a string - based in the field content.

[umeshdc]

@jochen and @jan Thanks for the quick help. This helped to fix my problem.