# Graylog Extraktor with lookup table

**URL:** <https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003>\
**Category:** Graylog Central (peer support)\
**Tags:** pipeline-rules\
**Created:** [April 17, 2018, 11:19am UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003 "2018-04-17T11:19:01Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![juemue](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@juemue](https://community.graylog.org/u/juemue)\
**Post date:** [April 17, 2018, 11:19am UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003/1 "2018-04-17T11:19:01Z")

</div>

I have log messages with key value entries. That’s why I want to use “Copy Input” as an extractor. Unfortunately, a field is extracted that contains a string and the field already exists as a numeric field. That’s why I added a lookup table as the second converter to the extractor. But it doesn’t work.

Error message:  
{“type”:“mapper\_parsing\_exception”,“reason”:“failed to parse [level]”,“caused\_by”:{“type”:“number\_format\_exception”,“reason”:“For input string: “notice””}}

Excerpt from the lookup table

“Notice”,“5”  
““Notice””,“5”

The “test lookup” on the data adapter works with “notice”, “Notice”, notice and Notice.

How can I get this extractor working?

---

<div class="post-metadata">

**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\
**Post date:** [April 17, 2018, 11:54am UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003/2 "2018-04-17T11:54:06Z")

</div>

You could use one or more pipeline rules to lookup the values and force the “level” message field to be a number (or store the lookup result in a different message field).

---

<div class="post-metadata">

**Author:** ![juemue](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@juemue](https://community.graylog.org/u/juemue)\
**Post date:** [April 18, 2018, 8:07am UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003/3 "2018-04-18T08:07:02Z")

</div>

@Jochen

Thx, I tested a pipeline, but the key\_value function works differently than the key\_value converter

Example:

subject=“TOTAL: Automatische Buchungsvorschlaege Maerz 2018”

result with key\_value function:  
fm\_subject  
TOTAL:

result with key\_value Converter:  
subject  
TOTAL: Automatische Buchungsvorschlaege Maerz 2018

Pipeline rule:

```
rule "keyvalue"
when
    has_field("message")
then
    let msg = to_string($message.message);
    set_fields(
                fields: 
                        key_value(
                            value: to_string($message.message) 
                       
                            ),
                prefix: "fm_"
            );
end;

```

What can I do, so that I get the same result with the pipeline as with the converter

---

<div class="post-metadata">

**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\
**Post date:** [April 18, 2018, 1:57pm UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003/4 "2018-04-18T13:57:07Z")

</div>

Please post the complete configuration of the extractor and converter you’re using.

---

<div class="post-metadata">

**Author:** ![juemue](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@juemue](https://community.graylog.org/u/juemue)\
**Post date:** [April 20, 2018, 8:32am UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003/5 "2018-04-20T08:32:49Z")

</div>

@jochen

I have now only the Key Value Extractor (converter) active:

```
{
  "total": 1,
  "extractors": [
{
  "id": "249c1420-446e-11e8-8a4a-0242ac120005",
  "title": "Key Value Pairs",
  "type": "copy_input",
  "converters": [
    {
      "type": "tokenizer",
      "config": {}
    }
  ],
  "order": 0,
  "exceptions": 0,
  "metrics": {
    "total": {
      "time": {
        "min": 0,
        "max": 0,
        "mean": 0,
        "std_dev": 0,
        "95th_percentile": 0,
        "98th_percentile": 0,
        "99th_percentile": 0
      },
      "rate": {
        "total": 0,
        "mean": 0,
        "one_minute": 0,
        "five_minute": 0,
        "fifteen_minute": 0
      },
      "duration_unit": "microseconds",
      "rate_unit": "events/second"
    },
    "condition": {
      "time": {
        "min": 0,
        "max": 0,
        "mean": 0,
        "std_dev": 0,
        "95th_percentile": 0,
        "98th_percentile": 0,
        "99th_percentile": 0
      },
      "rate": {
        "total": 0,
        "mean": 0,
        "one_minute": 0,
        "five_minute": 0,
        "fifteen_minute": 0
      },
      "duration_unit": "microseconds",
      "rate_unit": "events/second"
    },
    "execution": {
      "time": {
        "min": 0,
        "max": 0,
        "mean": 0,
        "std_dev": 0,
        "95th_percentile": 0,
        "98th_percentile": 0,
        "99th_percentile": 0
      },
      "rate": {
        "total": 0,
        "mean": 0,
        "one_minute": 0,
        "five_minute": 0,
        "fifteen_minute": 0
      },
      "duration_unit": "microseconds",
      "rate_unit": "events/second"
    },
    "converters": {
      "time": {
        "min": 0,
        "max": 0,
        "mean": 0,
        "std_dev": 0,
        "95th_percentile": 0,
        "98th_percentile": 0,
        "99th_percentile": 0
      },
      "rate": {
        "total": 0,
        "mean": 0,
        "one_minute": 0,
        "five_minute": 0,
        "fifteen_minute": 0
      },
      "duration_unit": "microseconds",
      "rate_unit": "events/second"
    },
    "condition_hits": 0,
    "condition_misses": 0
  },
  "cursor_strategy": "copy",
  "source_field": "message",
  "target_field": "message",
  "extractor_config": {},
  "creator_user_id": "***",
  "condition_type": "none",
  "condition_value": "",
  "converter_exceptions": 0
}
  ]
}

```

With this extractor, the subject field is extracted correctly.  
Unfortunately, not with the above pipeline

---

<div class="post-metadata">

**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\
**Post date:** [April 20, 2018, 8:59am UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003/6 "2018-04-20T08:59:38Z")

</div>

I don’t see any Lookup Table Extractor there.

---

<div class="post-metadata">

**Author:** ![juemue](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@juemue](https://community.graylog.org/u/juemue)\
**Post date:** [April 20, 2018, 9:10am UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003/7 "2018-04-20T09:10:55Z")

</div>

Where can I find these in the API . I used this endpoint:

api/system/inputs/5ad5b9a2adbe1d0001d309bc/extractors

This is the configuration in the web ui

 ![grafik](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/6/64795227afa526ebc19a5e00ac38a99a4ae3b1a1.png)

---

<div class="post-metadata">

**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\
**Post date:** [April 20, 2018, 9:24am UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003/8 "2018-04-20T09:24:50Z")

</div>

As you can see, that’s a Copy Input extractor with a Key-Value converter. You’ve mentioned a Lookup Table converter in your first post.

---

<div class="post-metadata">

**Author:** ![juemue](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@juemue](https://community.graylog.org/u/juemue)\
**Post date:** [April 20, 2018, 9:33am UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003/9 "2018-04-20T09:33:33Z")

</div>

That’s right. But you said I should use a pipeline instead of key-value converter with lookup table. And my problem is that the pipeline function does not have the same result as the key value converter.

The own lookup table is currently not the problem because the key value function in the pipeline does not work correctly at all

---

<div class="post-metadata">

**Author:** ![juemue](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@juemue](https://community.graylog.org/u/juemue)\
**Post date:** [April 20, 2018, 9:39am UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003/10 "2018-04-20T09:39:44Z")

</div>

This is the extractor configuration with key value converter and lookup table, which causes the error from the first post

```
{
  "total": 1,
  "extractors": [
    {
      "id": "639a36f0-447a-11e8-87c5-0242ac120005",
      "title": "Key Value Pairs ",
      "type": "copy_input",
      "converters": [
        {
          "type": "tokenizer",
          "config": {}
        },
        {
          "type": "lookup_table",
          "config": {
            "lookup_table_name": "Syslog Priority"
          }
        }
      ],
      "order": 0,
      "exceptions": 0,
      "metrics": {
        "total": {
          "time": {
            "min": 41,
            "max": 2324,
            "mean": 99,
            "std_dev": 92,
            "95th_percentile": 172,
            "98th_percentile": 258,
            "99th_percentile": 372
          },
          "rate": {
            "total": 565,
            "mean": 0.32988226143691796,
            "one_minute": 1.520866582345318e-11,
            "five_minute": 0.0179860647958941,
            "fifteen_minute": 0.5500309332144373
          },
          "duration_unit": "microseconds",
          "rate_unit": "events/second"
        },
        "condition": {
          "time": {
            "min": 0,
            "max": 46,
            "mean": 1,
            "std_dev": 2,
            "95th_percentile": 3,
            "98th_percentile": 4,
            "99th_percentile": 5
          },
          "rate": {
            "total": 565,
            "mean": 0.32988155049866286,
            "one_minute": 1.520941416473432e-11,
            "five_minute": 0.017986260218016272,
            "fifteen_minute": 0.5500318719965978
          },
          "duration_unit": "microseconds",
          "rate_unit": "events/second"
        },
        "execution": {
          "time": {
            "min": 37,
            "max": 2300,
            "mean": 89,
            "std_dev": 90,
            "95th_percentile": 157,
            "98th_percentile": 231,
            "99th_percentile": 356
          },
          "rate": {
            "total": 565,
            "mean": 0.3298810045319338,
            "one_minute": 1.520941416473432e-11,
            "five_minute": 0.017986260218016272,
            "fifteen_minute": 0.5500318719965978
          },
          "duration_unit": "microseconds",
          "rate_unit": "events/second"
        },
        "converters": {
          "time": {
            "min": 28,
            "max": 640,
            "mean": 62,
            "std_dev": 43,
            "95th_percentile": 110,
            "98th_percentile": 160,
            "99th_percentile": 224
          },
          "rate": {
            "total": 565,
            "mean": 0.32988047786937535,
            "one_minute": 1.520941416473432e-11,
            "five_minute": 0.017986260218016272,
            "fifteen_minute": 0.5500318719965978
          },
          "duration_unit": "microseconds",
          "rate_unit": "events/second"
        },
        "condition_hits": 0,
        "condition_misses": 0
      },
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "mesage",
      "extractor_config": {},
      "creator_user_id": "***",
      "condition_type": "none",
      "condition_value": "",
      "converter_exceptions": 0
    }
  ]
}
```

---

<div class="post-metadata">

**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\
**Post date:** [April 20, 2018, 9:53am UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003/11 "2018-04-20T09:53:17Z")

</div>

> [@juemue](#):
>
> And my problem is that the pipeline function does not have the same result as the key value converter.

What’s the difference given some example messages?

---

<div class="post-metadata">

**Author:** ![juemue](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@juemue](https://community.graylog.org/u/juemue)\
**Post date:** [April 20, 2018, 10:02am UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003/12 "2018-04-20T10:02:01Z")

</div>

subject=“TOTAL: Automatische Buchungsvorschlaege Maerz 2018”

result with key\_value function:  
fm\_subject  
TOTAL:

result with key\_value Converter:  
subject  
TOTAL: Automatische Buchungsvorschlaege Maerz 2018

---

<div class="post-metadata">

**Author:** ![jochen](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/jochen/32/8_2.png) [@jochen](https://community.graylog.org/u/jochen)\
**Post date:** [April 20, 2018, 10:17am UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003/13 "2018-04-20T10:17:00Z")

</div>

Please create a bug report at [https://github.com/Graylog2/graylog2-server/issues](https://github.com/Graylog2/graylog2-server/issues).

---

<div class="post-metadata">

**Author:** ![juemue](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@juemue](https://community.graylog.org/u/juemue)\
**Post date:** [April 23, 2018, 11:30am UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003/14 "2018-04-23T11:30:43Z")

</div>

Issue open

> <https://github.com/Graylog2/graylog2-server/issues/4751>

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [May 7, 2018, 11:30am UTC](https://community.graylog.org/t/graylog-extraktor-with-lookup-table/5003/15 "2018-05-07T11:30:54Z")

</div>

Closed on May 7, 2018, 11:30am UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.