How to rename nested field?

I have logs that under message key container level and it is the name of the log level. This causes issues when I’m trying to parse message to json due to "failed to parse field [level] of type [long] in document". From a bit of googling, looks like it’s due to level being reserved by graylog and defined ES index that says that level is a number.

I created a rule to try and rename the field from level to level_name, but it doesn’t seem to work:

rule "rename event level"
    rename_field("level", "level_name");

Log message are sent using graypy log handler and have message field, which contains json log params from my system.

Example message object:

"event": "pidbox received method enable_events() [reply_to:None ticket:None]", "level": "debug", "timestamp": "2020-07-09T13:46:12.923189Z", "funcName": "dispatch", "thread": 139943025719104, "pathname": "/usr/local/lib/python3.7/site-packages/kombu/", "lineno": 101, "process": 19}
  1. How do you use json extraction? Using extractor or pipeline rule?
  2. The simplest way will be probably to change field level in graypy if it’s possible
  3. If not, maybe use pipeline rule to rename text level to another one (level_app) and apply json extraction afterwards.
rule "replace level to level_app"
    let level_rename = regex_replace("\"level\":", to_string($message.message), "\"level_app\":");
    set_field("message", level_rename);
    let json = parse_json(to_string(level_rename));
    let map = to_map(json);

I think I understand what was the issue. GELF has level by itself and it is a number. So when I was using json extractor to extract key-values from message it was extracting level from message to upper level. Problem is, elasticsearch index already registered level and detected that is is number and extracted level value is a string.

Not sure if it’s the best solution, but I just added a prefex in json extractor and so now level was extracted to <prefex>level and not overriding previously indexed GELF level.

Yes, that is also one option.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.