Graylog Extraktor with lookup table

Graylog Central (peer support)
1

I have log messages with key value entries. That’s why I want to use “Copy Input” as an extractor. Unfortunately, a field is extracted that contains a string and the field already exists as a numeric field. That’s why I added a lookup table as the second converter to the extractor. But it doesn’t work.

Error message:
{“type”:“mapper_parsing_exception”,“reason”:“failed to parse [level]”,“caused_by”:{“type”:“number_format_exception”,“reason”:“For input string: “notice””}}

Excerpt from the lookup table

“Notice”,“5”
““Notice””,“5”

The “test lookup” on the data adapter works with “notice”, “Notice”, notice and Notice.

How can I get this extractor working?

2

You could use one or more pipeline rules to lookup the values and force the “level” message field to be a number (or store the lookup result in a different message field).

3

@Jochen

Thx, I tested a pipeline, but the key_value function works differently than the key_value converter

Example:

subject=“TOTAL: Automatische Buchungsvorschlaege Maerz 2018”

result with key_value function:
fm_subject
TOTAL:

result with key_value Converter:
subject
TOTAL: Automatische Buchungsvorschlaege Maerz 2018

Pipeline rule:

rule "keyvalue"
when
    has_field("message")
then
    let msg = to_string($message.message);
    set_fields(
                fields: 
                        key_value(
                            value: to_string($message.message) 
                       
                            ),
                prefix: "fm_"
            );
end;

What can I do, so that I get the same result with the pipeline as with the converter

4

Please post the complete configuration of the extractor and converter you’re using.

5

@jochen

I have now only the Key Value Extractor (converter) active:

{
  "total": 1,
  "extractors": [
{
  "id": "249c1420-446e-11e8-8a4a-0242ac120005",
  "title": "Key Value Pairs",
  "type": "copy_input",
  "converters": [
    {
      "type": "tokenizer",
      "config": {}
    }
  ],
  "order": 0,
  "exceptions": 0,
  "metrics": {
    "total": {
      "time": {
        "min": 0,
        "max": 0,
        "mean": 0,
        "std_dev": 0,
        "95th_percentile": 0,
        "98th_percentile": 0,
        "99th_percentile": 0
      },
      "rate": {
        "total": 0,
        "mean": 0,
        "one_minute": 0,
        "five_minute": 0,
        "fifteen_minute": 0
      },
      "duration_unit": "microseconds",
      "rate_unit": "events/second"
    },
    "condition": {
      "time": {
        "min": 0,
        "max": 0,
        "mean": 0,
        "std_dev": 0,
        "95th_percentile": 0,
        "98th_percentile": 0,
        "99th_percentile": 0
      },
      "rate": {
        "total": 0,
        "mean": 0,
        "one_minute": 0,
        "five_minute": 0,
        "fifteen_minute": 0
      },
      "duration_unit": "microseconds",
      "rate_unit": "events/second"
    },
    "execution": {
      "time": {
        "min": 0,
        "max": 0,
        "mean": 0,
        "std_dev": 0,
        "95th_percentile": 0,
        "98th_percentile": 0,
        "99th_percentile": 0
      },
      "rate": {
        "total": 0,
        "mean": 0,
        "one_minute": 0,
        "five_minute": 0,
        "fifteen_minute": 0
      },
      "duration_unit": "microseconds",
      "rate_unit": "events/second"
    },
    "converters": {
      "time": {
        "min": 0,
        "max": 0,
        "mean": 0,
        "std_dev": 0,
        "95th_percentile": 0,
        "98th_percentile": 0,
        "99th_percentile": 0
      },
      "rate": {
        "total": 0,
        "mean": 0,
        "one_minute": 0,
        "five_minute": 0,
        "fifteen_minute": 0
      },
      "duration_unit": "microseconds",
      "rate_unit": "events/second"
    },
    "condition_hits": 0,
    "condition_misses": 0
  },
  "cursor_strategy": "copy",
  "source_field": "message",
  "target_field": "message",
  "extractor_config": {},
  "creator_user_id": "***",
  "condition_type": "none",
  "condition_value": "",
  "converter_exceptions": 0
}
  ]
}

With this extractor, the subject field is extracted correctly.
Unfortunately, not with the above pipeline

6

I don’t see any Lookup Table Extractor there.

7

Where can I find these in the API . I used this endpoint:

api/system/inputs/5ad5b9a2adbe1d0001d309bc/extractors

This is the configuration in the web ui

grafik
grafik.png956×725 36.4 KB

8

As you can see, that’s a Copy Input extractor with a Key-Value converter. You’ve mentioned a Lookup Table converter in your first post.

9

That’s right. But you said I should use a pipeline instead of key-value converter with lookup table. And my problem is that the pipeline function does not have the same result as the key value converter.

The own lookup table is currently not the problem because the key value function in the pipeline does not work correctly at all

10

This is the extractor configuration with key value converter and lookup table, which causes the error from the first post

{
  "total": 1,
  "extractors": [
    {
      "id": "639a36f0-447a-11e8-87c5-0242ac120005",
      "title": "Key Value Pairs ",
      "type": "copy_input",
      "converters": [
        {
          "type": "tokenizer",
          "config": {}
        },
        {
          "type": "lookup_table",
          "config": {
            "lookup_table_name": "Syslog Priority"
          }
        }
      ],
      "order": 0,
      "exceptions": 0,
      "metrics": {
        "total": {
          "time": {
            "min": 41,
            "max": 2324,
            "mean": 99,
            "std_dev": 92,
            "95th_percentile": 172,
            "98th_percentile": 258,
            "99th_percentile": 372
          },
          "rate": {
            "total": 565,
            "mean": 0.32988226143691796,
            "one_minute": 1.520866582345318e-11,
            "five_minute": 0.0179860647958941,
            "fifteen_minute": 0.5500309332144373
          },
          "duration_unit": "microseconds",
          "rate_unit": "events/second"
        },
        "condition": {
          "time": {
            "min": 0,
            "max": 46,
            "mean": 1,
            "std_dev": 2,
            "95th_percentile": 3,
            "98th_percentile": 4,
            "99th_percentile": 5
          },
          "rate": {
            "total": 565,
            "mean": 0.32988155049866286,
            "one_minute": 1.520941416473432e-11,
            "five_minute": 0.017986260218016272,
            "fifteen_minute": 0.5500318719965978
          },
          "duration_unit": "microseconds",
          "rate_unit": "events/second"
        },
        "execution": {
          "time": {
            "min": 37,
            "max": 2300,
            "mean": 89,
            "std_dev": 90,
            "95th_percentile": 157,
            "98th_percentile": 231,
            "99th_percentile": 356
          },
          "rate": {
            "total": 565,
            "mean": 0.3298810045319338,
            "one_minute": 1.520941416473432e-11,
            "five_minute": 0.017986260218016272,
            "fifteen_minute": 0.5500318719965978
          },
          "duration_unit": "microseconds",
          "rate_unit": "events/second"
        },
        "converters": {
          "time": {
            "min": 28,
            "max": 640,
            "mean": 62,
            "std_dev": 43,
            "95th_percentile": 110,
            "98th_percentile": 160,
            "99th_percentile": 224
          },
          "rate": {
            "total": 565,
            "mean": 0.32988047786937535,
            "one_minute": 1.520941416473432e-11,
            "five_minute": 0.017986260218016272,
            "fifteen_minute": 0.5500318719965978
          },
          "duration_unit": "microseconds",
          "rate_unit": "events/second"
        },
        "condition_hits": 0,
        "condition_misses": 0
      },
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "mesage",
      "extractor_config": {},
      "creator_user_id": "***",
      "condition_type": "none",
      "condition_value": "",
      "converter_exceptions": 0
    }
  ]
}
11

What’s the difference given some example messages?

12

subject=“TOTAL: Automatische Buchungsvorschlaege Maerz 2018”

result with key_value function:
fm_subject
TOTAL:

result with key_value Converter:
subject
TOTAL: Automatische Buchungsvorschlaege Maerz 2018

13

Please create a bug report at https://github.com/Graylog2/graylog2-server/issues.

14

Issue open

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.