Howto figure out the cause of a field type conflict

(Jason Haar) #1

Hi there

I have a “dst_port” field defined (numeric) via extractors. Works great. However, something is pushing dst_port values in as another type - causing ES errors like

[graylog_7009][1] failed to execute bulk item (index) BulkShardRequest [[graylog_7009][1]] containing [115] requests
java.lang.IllegalArgumentException: mapper [dst_port] of different type, current_type [long], merged_type [keyword]

How can I hunt down the offender? We don’t control all the inputs, so someone is probably pushing a GELF feed into us with that defined incorrectly, but how do I find it?



(Jan Doberstein) #2

I can think of a processing pipeline that checks the field dst_port not beeing a number and then rename it - or write a debug log entry in the graylog server.log that contains the message source (and then delete the message).

Something like the following (this is untested!)

rule "dst_port_not_long"
    has_field("dst_port") AND
    to_long($message.dst_port) == 0
   rename_field("dst_port", "dst_port_nn")

How to handle elasticsearch mapping error due to different data type
(Jochen) #3

Related PR for the processing pipelines plugin:

(123dev) #4

You can also define the type in custom index mapping
It would solve your problem, but am not sure if the entire message is dropped by ES or just that field, I fear the former.

Jan’s solution is cleanest, I’m thinking we should adopt it. :slight_smile:

(system) #5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.