Extractor not working properly for 'level' field

[Gioo]

I’m facing an issue with extracting the “level” field. My goal is to convert the “level” field from its long value (0, 1, 2, etc.) to its string label (Emergency, Alert, Critical, etc.). To achieve this, I intend to use a lookup table that I have already created following the documentation, and it seems to work correctly. However, the problem arises when I attempt to use this lookup table in an extractor, as the extractor appears to have no effect.

In an attempt to simplify the problem, I’ve experimented with extractors in less complex scenarios, such as a straightforward field copy. The perplexing aspect is that the extractors seem to work for all fields EXCEPT the “level” field. I’m struggling to comprehend why this is happening, given that I have configured them all in the exact same way.

Let me know what I might be doing wrong.

[s0p4L1N]

I will give you how I do it, and I do not use extractors, only pipeline rules (extractors will be deprecated in a future)

Requirements

• Create a stream and you should have your source (vmware or firewall or etc) logs in this steam.

PART I - LOOKUP TABLE

1. First step

I have my CSV file located in /srv/RFC_log_level.csv on my Graylog host.

"Level";"SEVERITY"
"0";"Emergency"
"1";"Alert"
"2";"Critical"
"3";"Error"
"4";"Warning"
"5";"Notice"
"6";"Informational"
"7";"Debug"

2. Second step

Then on System > Lookup Tables > Data Adapters

You have to configure all the needed info.

3. Third step

On System > Lookup Tables > Cache

You just have to create a cache.

4. Fourth step

On System > Lookup Table

Create your Lookup Table referencing the previous Data Adapters and Cache.

If you test the lookup table, it works

But you are not done, because you need to tell to graylog to take action on the fields and use the lookup table if certain condition is valid.
Extractors can not do this, only pipeline rule.

PART II - PIPELINE RULE

1. Step 1

Go to System > Pipeline > Manage rules > Create Rule

Here’s my pipeline rule for my source (VMWARE)

rule "Syslog - VMWARE - log_level lookup"

when
  has_field("level")
  then
   let new_level_severity = lookup_value("rfc_log_level", to_string($message.level));
   set_field("level_severity", new_level_severity);

end

Where:

• has_field("level")

  • tells graylog to check if the field level is present in the log

• let new_level_severity =

  • you just declare a variable

• lookup_value("rfc_log_level", to_string($message.level));

  • content of the variable where you use the lookup_value function, it will use the Lookup table, and check the matching value from first row of your CSV with the value of the level field.

• rfc_log_level is the lookup table name you just created before in PART 1 - Step 4

• set_field("level_severity", new_level_severity);

  • Tells to graylog to create a new field level_severity with the value matched from the second raw of your CSV file by calling the previous variable.

2. Step 2

Go to System > Pipeline > Add New Pipeline

You have created the previous rule, but now it does nothing. You need to create the Pipeline and connect the pipeline to your Stream and add the Rule to a Stage.

You can see the results by look at your Stream or test with a message in System > Pipeline > Simulator.

[Gioo]

Thank you so much! Your solution worked perfectly for me.