Hi, I’m using the “old” AWS logs plugin — not the one from the recent integrations plugin in 3.1.1, I’m running Graylog 3.1.0 — to collect logs from all my log groups.
Some of these log groups emit deeply nested JSON messages and I want to parse them in a pipeline using parse_json
to avoid having the really long a_b_c_d_e_f
keys that the regular json extractor emits, just the top level keys are fine to me. Also to avoid going over the 1000 fields ES limit that Graylog uses by default.
The relevant pipeline rule looks like this:
rule "extract-json"
when
starts_with(to_string($message.message), "{") && ends_with(to_string($message.message), "}")
then
let json = parse_json(to_string($message.message));
let map = to_map(json);
set_fields(map);
end
Problem is some of these json messages contain a timestamp
field which, as expected from being regular json, is a string. And then Graylog can’t index the message properly because the timestamp field is a string and not a date.
Is it possible to either remove the timestamp
key from the map or properly parse it into a date object without losing the other fields? e.g.:
// examples
map["timestamp"] = parse_date(map["timestamp"]);
remove_key(map, "timestamp");
Failing that, is it possible to make the json extractor limit extraction to the top level keys or up to a certain depth?