Extractor not working properly for 'level' field

[s0p4L1N]

I will give you how I do it, and I do not use extractors, only pipeline rules (extractors will be deprecated in a future)

Requirements

• Create a stream and you should have your source (vmware or firewall or etc) logs in this steam.

PART I - LOOKUP TABLE

1. First step

I have my CSV file located in /srv/RFC_log_level.csv on my Graylog host.

"Level";"SEVERITY"
"0";"Emergency"
"1";"Alert"
"2";"Critical"
"3";"Error"
"4";"Warning"
"5";"Notice"
"6";"Informational"
"7";"Debug"

2. Second step

Then on System > Lookup Tables > Data Adapters

You have to configure all the needed info.

3. Third step

On System > Lookup Tables > Cache

You just have to create a cache.

4. Fourth step

On System > Lookup Table

Create your Lookup Table referencing the previous Data Adapters and Cache.

If you test the lookup table, it works

But you are not done, because you need to tell to graylog to take action on the fields and use the lookup table if certain condition is valid.
Extractors can not do this, only pipeline rule.

PART II - PIPELINE RULE

1. Step 1

Go to System > Pipeline > Manage rules > Create Rule

Here’s my pipeline rule for my source (VMWARE)

rule "Syslog - VMWARE - log_level lookup"

when
  has_field("level")
  then
   let new_level_severity = lookup_value("rfc_log_level", to_string($message.level));
   set_field("level_severity", new_level_severity);

end

Where:

• has_field("level")

  • tells graylog to check if the field level is present in the log

• let new_level_severity =

  • you just declare a variable

• lookup_value("rfc_log_level", to_string($message.level));

  • content of the variable where you use the lookup_value function, it will use the Lookup table, and check the matching value from first row of your CSV with the value of the level field.

• rfc_log_level is the lookup table name you just created before in PART 1 - Step 4

• set_field("level_severity", new_level_severity);

  • Tells to graylog to create a new field level_severity with the value matched from the second raw of your CSV file by calling the previous variable.

2. Step 2

Go to System > Pipeline > Add New Pipeline

You have created the previous rule, but now it does nothing. You need to create the Pipeline and connect the pipeline to your Stream and add the Rule to a Stage.

You can see the results by look at your Stream or test with a message in System > Pipeline > Simulator.