Removing key from pipeline map

Graylog Central (peer support)
1

Hi, I’m using the “old” AWS logs plugin — not the one from the recent integrations plugin in 3.1.1, I’m running Graylog 3.1.0 — to collect logs from all my log groups.

Some of these log groups emit deeply nested JSON messages and I want to parse them in a pipeline using parse_json to avoid having the really long a_b_c_d_e_f keys that the regular json extractor emits, just the top level keys are fine to me. Also to avoid going over the 1000 fields ES limit that Graylog uses by default.

The relevant pipeline rule looks like this:

rule "extract-json"
when
    starts_with(to_string($message.message), "{") && ends_with(to_string($message.message), "}")
then
    let json = parse_json(to_string($message.message));
    let map = to_map(json);
    set_fields(map);
end

Problem is some of these json messages contain a timestamp field which, as expected from being regular json, is a string. And then Graylog can’t index the message properly because the timestamp field is a string and not a date.

Is it possible to either remove the timestamp key from the map or properly parse it into a date object without losing the other fields? e.g.:

// examples
map["timestamp"] = parse_date(map["timestamp"]);
remove_key(map, "timestamp");

Failing that, is it possible to make the json extractor limit extraction to the top level keys or up to a certain depth?

2

both requests are valid and both should be a feature request over at github:

3

Thank you, just created the following two feature requests:


4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.