Grok Pipeline Output

Graylog Central (peer support)
1

I have the below pipeline processor rule that allows me to extract the hour from the TimeStamp field.

rule "HourExtractor"
when
has_field("TimeStamp")
then
let Time = to_string($message.TimeStamp);
let Hour = grok("%{DATA:UNWANTED}T%{NUMBER:Hour}%{DATA:UNWANTED}", Time, true);
set_field("Hour", to_string(Hour));
end

It almost works as expected however when simulating with the following message:

(2018-03-16T11:13:43) tcp-udp-proxy[2212]: 0x10cca270-570567 281846384:570567: new connection 49: xxx.xxx.xxx.xxx:xxxxx → xxx.xxx.xxx.xxx:xxx [A t] {N} | -1: xxx.xxx.xxx.xxx:xxxxx → xxx.xxx.xxx.xxx:xxx [!B] {N}[U!P]

I get the output:
Added fields:

Hour
{Hour=11}
TimeStamp
2018-03-16T11:22:46.466Z

I would however like the output be 11 instead of {Hour=11}, anyone know how to do this?

Cheers,

G

2

If you’re trying to get the hour of the “timestamp” field (i. e. a real DateTime object), you could use the following rule:

rule "hour-extractor"
when
  true // all messages have the "timestamp" field by default
then
  let hour = $message.timestamp.hourOfDay;
  set_field("hour", hour);
end

As for the grok() function, it returns a map, so you could use set_fields() to assign all results to the current message:

rule "HourExtractor"
when
  has_field("TimeStamp")
then
  let Time = to_string($message.TimeStamp);
  let Hour = grok("%{DATA:UNWANTED}T%{NUMBER:Hour}%{DATA:UNWANTED}", Time, true);
  set_fields(Hour);
end
3

We parse into camel case fields so TimeStamp is our normalized timestamp field, Windows Event logs come with the field timestamp and we had to use pipelines to copy the timestamp field to the TimeStamp field as using a copy extractor didn’t work. Using set fields worked perfectly, thank you for your help.

4

You just might want to store the hour as a numeric field if you plan to use range queries on it later (e. g. Hour:[9 TO 18].

5

Hi @jochen,

We are only using a stream that has rule hour matches regex (19|20|21|22|23|00|01|02|03|04|05|06) This allows us to see out of hours activity. However I will bear in mind your comment for future.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.