Set_fields Usage?

[danmassa7]

I have syslog message roughly compatible with the syslog RFC and I’m trying to use pipelines and rules to get better control over parsing these messages. But I’m not a programmer so there are things that are tripping me up.

I have the following to do initial processing of the message:

rule "raw-process-exp"
when 
    true
then    
    // Raw processing Tests 
    let extract = grok(
        pattern: "<%{NONNEGINT:syslog_pri}>", 
        value: to_string($message.message),
        only_named_captures: true
        );  
    let syslogpriexanded = expand_syslog_priority(extract.syslog_pri);
    set_field("syslog_severity", syslogpriexanded);
    set_fields(extract);
    set_field("record_type", "EXPERIMENTAL LOG PROCESSING");
 end

What gets put in the syslog_severity field is:
{"level":3,"facility":17}

I guess that’s what they call a map or a something? I want to actually pull out the “3” and put it in the syslog_severity field. I’ve tried a couple of things but haven’t figured it out yet.

Or maybe there’s a way to use the set_fields function to automatically add a field named “level” and set it to “3” and add a field called “facility” and set it to 17? That would be cool. Haven’t figure that out either.

Is there a super-simple example somewhere on how to use the set_fields function? How do I learn the ins and outs of using these functions?

Thanks!

Operating system information

centos-linux-release-8.3-1.2011.el8.noarch

Package versions

• Graylog 4.0.7
• MongoDB 4.2.14
• Elasticsearch 7.10.2

[tmacgbay]

Based on what you are saying, this line:

let syslogpriexanded = expand_syslog_priority(extract.syslog_pri);

Is giving you the map you want of: {"level":3,"facility":17}

so you would set_fields() against that:

set_fields(syslogpriexanded);

which should get the results you wanted.

set_fields() is usually following a grok() and taking the resulting file-to-data mapping and creating a series of fields that grok() matched. You can put the debug() function in any rule and watch the Graylog server logs for the results. Here is an example rule using grok() and debug():

rule "bar-1-start"
when
    has_field("bar_event_1")                            &&
    contains(to_string($message.message),"bar session started ")
then

    let barLine = grok("%{BAR_1_START}",to_string($message.message), true); 
    set_fields(barLine);
    debug(concat("======= bar_event_1 is: ", to_string($message.bar_event_1) );
    debug(concat("======= message is: ", to_string($message.message) );
end

to watch Graylog server logs, use:

tail -f /var/log/graylog-server/server.log

[gsmith]

Hello,

If pipelines are not your cup of tea, you can use an extractor to get/make the field Level.

Example:

Grok Pattern

Name: Level
Pattern: (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))

Then use it in your extractor.

GROK: level=%{Level}

Output:

[danmassa7]

That was one of the first things I tried. I tried it with…

set_fields(extract);

…and it worked great. It seems the grok() function returns a map and the set_fields takes a map. I know this because I ran is_it_map(extract) and it returned TRUE.

But it doesn’t work with syslogpriexpanded(). It doesn’t even compile and just gives the tool-tip message…

Expected type Map for argument fields but found SyslogPriority in call to function set_fields in line 35 pos 4.

Another crazy thing: the output of the expand_syslogpriority() function is not a map. I don’t know what it is. I ran it against:

is_it_map()
is_it_collection()
is_it_list()
is_it_json()
is_it_string()

Everything came up FALSE. What the output of expand_syslogpriority()? Weird. It would be helpful if the documentation indicated the output datatype of expand_syslogpriority(). It doesn’t.

[danmassa7]

@gsmith Your idea is a good one and we are already doing grok patterns on extractors. But I did notice that we are doing the same grok patterns multiple type across inputs. The pipeline/rule system seems useful for code reusability, so I figured things we be easier to implement there.
We are also finding that we need to do some more advanced post-processing. E.g., change a string to a long integer or reformate a date. We’re trying to bring our syslog processing to the next level!

[tmacgbay]

Looks like Graylog needs us to be more specific on the set_fields() Here is an example I found in another forum post that @shoothub put up:

rule "Cisco FirePower priority parsing"
when
  contains(to_string($message.message), "%FTD", true)
then
  set_fields(grok(pattern: "<%{NONNEGINT:syslog_pri:int}>", value: to_string($message.message), only_named_captures: true)); 
  let priority = expand_syslog_priority(to_long($message.syslog_pri));
  set_fields({facility: priority.facility, level: priority.level });
end

(If this is the answer, please mark it as so for future searchers!)

[danmassa7]

I ended up doing this the long way. Or maybe this is the proper way? If there is a more elegant way to do this I’m open to it.

First I pull out my stuff:

    let extract = grok(
        pattern: "<%{NONNEGINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_time} %{YEAR:year}",
        //pattern: "<%{NONNEGINT:syslog_pri}>", 
        value: to_string($message.message),
        only_named_captures: true
        );

I process the priority field.

let syslog_sev_fac = expand_syslog_priority(extract.syslog_pri);

I pull it into variables.

    let syslog_sev = to_long(syslog_sev_fac.level);
    let syslog_fac = to_long(syslog_sev_fac.facility);
    let syslog_severity = syslog_level(syslog_sev);
    let syslog_facility = syslog_facility(syslog_fac);

Then set the fields.

    set_field("severity", syslog_sev);
    set_field("severity_desc", syslog_severity);
    set_field("facility", syslog_fac);
    set_field("facility_desc", syslog_facility);

This works. I just was thinking things could go in easier with the set_fields function.

If someone has a better way I’m all ears!
Thanks.