Gork extractor not matching any message

Graylog Tech Challenges
1

I created the following extractor:

{
  "extractors": [
    {
      "title": "Bind query extractor with view",
      "extractor_type": "grok",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "",
      "extractor_config": {
        "grok_pattern": "client %{DATA:client_cookie} %{IPORHOST:client_ip}#%{POSINT:client_port} (%{DATA}): %{BindViewName} query: %{GREEDYDATA:query}",
        "named_captures_only": false
      },
      "condition_type": "none",
      "condition_value": ""
    }
  ],
  "version": "4.0.8"
}

Metric details say that it haven’t matched anything, but there are messages like this one:

<30>1 2021-06-19T17:10:55.646107-04:00 bind01 named 38409 - -  client @0x7f9dcc4d5b50 172.16.10.39#43918 (www.amazon.com): view normal: query: www.amazon.com IN AAAA + (172.16.15.48)

client @0x7f9dcc4d5b50 172.16.10.39#43918 (www.amazon.com): view normal: query: www.amazon.com IN AAAA + (172.16.15.48)

I put everything on the message and apply my extractor to it.

Testing the extractor against these messages do extract them, but otherwise it’s like the extractor didn’t exist in the input.

2

Hello, braiam,

I've noticed your post hasn't been responded to.  Moving it to Daily Challenges for perhaps a helpful response.  In the meantime, here's a previous article on your topic.  Let me know if it helps.
3

In that specific case, the preview doesn’t show any matched field. In my case, there are matched fields, but it isn’t hit by the input.

4

Hi @braiam
I’ve tested your grok and didn’t work for obvious reasons:

  1. Your grok pattern %{BindViewName} is probably not defined, so graylog can’t process extractor at all
  2. Another problem is that you didn’t enabled Named captures only in extractor definition. So %{DATA} in your grok is saved as generic field DATA, which is propably not desired.
  3. If there is another problem with extractor, always check graylog server logs file. If graylog didn’t process extractor for any reason, there should be error or warning in logs file:
    sudo tail -f /var/log/graylog-server/server.log
5

There’s some problems with that theories:

  1. If it was not defined, Graylog would complain about it loudly on “Try against example” with “We were not able to run the grok extraction because of the following error: No definition for key ‘DdATA’ found, aborting” for example. I changed it to plain “DATA:view”, no dice.
  2. While named_captures_only is disabled, it shouldn’t effect whenever it is matched or not, just that the field name would be something like “DATA”.
  3. The extractor metrics says “0 total invocations since boot, averages: 0, 0, 0.”, which means that it doesn’t even fire when the input retrieves a new message. There’s no error about the pattern, since the pattern is never tried.

As I said, it should be firing since the condition is “Always try to extract” even if it doesn’t match anything, and the “Try against example” correctly extracts the data.

6

So, a development: I created a extractor that simply copies message to another field, same “Always try to extract”, still not firing. I think I’ve hit a bug.

7

I’ve tried your grok and worked fine for me in versions 4.0.7 and 4.1.

a. I don’t think that there is a bug in such essential feature as simple extractor.
b. Check if you didn’t disabled Message Filter Chain by accident in System - Configurations in section Message Processors Configuration.
c. What’s your Message Processors Configuration order? Message Filter Chain is after or before Pipeline Processor
d. Do you use any pipeline rules, which should collide with extractor?
e. What type of input do you use? Syslog TCP or UDP or another one?
f. Do you have same problem with other extractors, or none of extractors works?

2 Likes
8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.