I have logs that under message key container level and it is the name of the log level. This causes issues when I’m trying to parse message to json due to "failed to parse field [level] of type [long] in document". From a bit of googling, looks like it’s due to level being reserved by graylog and defined ES index that says that level is a number.
I created a rule to try and rename the field from level to level_name, but it doesn’t seem to work:
rule "rename event level"
when
has_field("level")
then
rename_field("level", "level_name");
end
Log message are sent using graypy log handler and have message field, which contains json log params from my system.
How do you use json extraction? Using extractor or pipeline rule?
The simplest way will be probably to change field level in graypy if it’s possible
If not, maybe use pipeline rule to rename text level to another one (level_app) and apply json extraction afterwards.
rule "replace level to level_app"
when
has_field("message")
then
let level_rename = regex_replace("\"level\":", to_string($message.message), "\"level_app\":");
set_field("message", level_rename);
let json = parse_json(to_string(level_rename));
let map = to_map(json);
set_fields(map);
end
I think I understand what was the issue. GELF has level by itself and it is a number. So when I was using json extractor to extract key-values from message it was extracting level from message to upper level. Problem is, elasticsearch index already registered level and detected that is is number and extracted level value is a string.
Not sure if it’s the best solution, but I just added a prefex in json extractor and so now level was extracted to <prefex>level and not overriding previously indexed GELF level.
