Set_fields Usage?

I have syslog message roughly compatible with the syslog RFC and I’m trying to use pipelines and rules to get better control over parsing these messages. But I’m not a programmer so there are things that are tripping me up.

I have the following to do initial processing of the message:

rule "raw-process-exp"
    // Raw processing Tests 
    let extract = grok(
        pattern: "<%{NONNEGINT:syslog_pri}>", 
        value: to_string($message.message),
        only_named_captures: true
    let syslogpriexanded = expand_syslog_priority(extract.syslog_pri);
    set_field("syslog_severity", syslogpriexanded);
    set_field("record_type", "EXPERIMENTAL LOG PROCESSING");

What gets put in the syslog_severity field is:

I guess that’s what they call a map or a something? I want to actually pull out the “3” and put it in the syslog_severity field. I’ve tried a couple of things but haven’t figured it out yet.

Or maybe there’s a way to use the set_fields function to automatically add a field named “level” and set it to “3” and add a field called “facility” and set it to 17? That would be cool. Haven’t figure that out either.

Is there a super-simple example somewhere on how to use the set_fields function? How do I learn the ins and outs of using these functions?


Operating system information


Package versions

  • Graylog 4.0.7
  • MongoDB 4.2.14
  • Elasticsearch 7.10.2

Based on what you are saying, this line:

let syslogpriexanded = expand_syslog_priority(extract.syslog_pri);

Is giving you the map you want of: {"level":3,"facility":17}

so you would set_fields() against that:


which should get the results you wanted.

set_fields() is usually following a grok() and taking the resulting file-to-data mapping and creating a series of fields that grok() matched. You can put the debug() function in any rule and watch the Graylog server logs for the results. Here is an example rule using grok() and debug():

rule "bar-1-start"
    has_field("bar_event_1")                            &&
    contains(to_string($message.message),"bar session started ")

    let barLine = grok("%{BAR_1_START}",to_string($message.message), true); 
    debug(concat("======= bar_event_1 is: ", to_string($message.bar_event_1) );
    debug(concat("======= message is: ", to_string($message.message) );

to watch Graylog server logs, use:

tail -f /var/log/graylog-server/server.log


If pipelines are not your cup of tea, you can use an extractor to get/make the field Level.


Grok Pattern

Name: Level
Pattern: (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))

Then use it in your extractor.

GROK: level=%{Level}


That was one of the first things I tried. I tried it with…


…and it worked great. It seems the grok() function returns a map and the set_fields takes a map. I know this because I ran is_it_map(extract) and it returned TRUE.

But it doesn’t work with syslogpriexpanded(). It doesn’t even compile and just gives the tool-tip message…

Expected type Map for argument fields but found SyslogPriority in call to function set_fields in line 35 pos 4.

Another crazy thing: the output of the expand_syslogpriority() function is not a map. I don’t know what it is. I ran it against:


Everything came up FALSE. What the output of expand_syslogpriority()? Weird. It would be helpful if the documentation indicated the output datatype of expand_syslogpriority(). It doesn’t.

@gsmith Your idea is a good one and we are already doing grok patterns on extractors. But I did notice that we are doing the same grok patterns multiple type across inputs. The pipeline/rule system seems useful for code reusability, so I figured things we be easier to implement there.
We are also finding that we need to do some more advanced post-processing. E.g., change a string to a long integer or reformate a date. We’re trying to bring our syslog processing to the next level!

1 Like

Looks like Graylog needs us to be more specific on the set_fields() Here is an example I found in another forum post that @shoothub put up:

rule "Cisco FirePower priority parsing"
  contains(to_string($message.message), "%FTD", true)
  set_fields(grok(pattern: "<%{NONNEGINT:syslog_pri:int}>", value: to_string($message.message), only_named_captures: true)); 
  let priority = expand_syslog_priority(to_long($message.syslog_pri));
  set_fields({facility: priority.facility, level: priority.level });

(If this is the answer, please mark it as so for future searchers!)

1 Like

I ended up doing this the long way. Or maybe this is the proper way? If there is a more elegant way to do this I’m open to it.

First I pull out my stuff:

    let extract = grok(
        pattern: "<%{NONNEGINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_time} %{YEAR:year}",
        //pattern: "<%{NONNEGINT:syslog_pri}>", 
        value: to_string($message.message),
        only_named_captures: true

I process the priority field.

let syslog_sev_fac = expand_syslog_priority(extract.syslog_pri);

I pull it into variables.

    let syslog_sev = to_long(syslog_sev_fac.level);
    let syslog_fac = to_long(syslog_sev_fac.facility);
    let syslog_severity = syslog_level(syslog_sev);
    let syslog_facility = syslog_facility(syslog_fac);

Then set the fields.

    set_field("severity", syslog_sev);
    set_field("severity_desc", syslog_severity);
    set_field("facility", syslog_fac);
    set_field("facility_desc", syslog_facility);

This works. I just was thinking things could go in easier with the set_fields function.

If someone has a better way I’m all ears!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.