Howto figure out the cause of a field type conflict


(Jason Haar) #1

Hi there

I have a “dst_port” field defined (numeric) via extractors. Works great. However, something is pushing dst_port values in as another type - causing ES errors like

[graylog_7009][1] failed to execute bulk item (index) BulkShardRequest [[graylog_7009][1]] containing [115] requests
java.lang.IllegalArgumentException: mapper [dst_port] of different type, current_type [long], merged_type [keyword]

How can I hunt down the offender? We don’t control all the inputs, so someone is probably pushing a GELF feed into us with that defined incorrectly, but how do I find it?

Thanks

Jason


(Jan Doberstein) #2

I can think of a processing pipeline that checks the field dst_port not beeing a number and then rename it - or write a debug log entry in the graylog server.log that contains the message source (and then delete the message).

Something like the following (this is untested!)

rule "dst_port_not_long"
when
    has_field("dst_port") AND
    to_long($message.dst_port) == 0
then
   rename_field("dst_port", "dst_port_nn")
end

How to handle elasticsearch mapping error due to different data type
(Jochen) #3

Related PR for the processing pipelines plugin:


(123dev) #4

You can also define the type in custom index mapping
http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html
It would solve your problem, but am not sure if the entire message is dropped by ES or just that field, I fear the former.

Jan’s solution is cleanest, I’m thinking we should adopt it. :slight_smile:


(system) #5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.