Pipeline does not set new field while querying if message is generated during business hours

Hi Guys, I’m new to the forum. I hope I won’t do anything wrong.
I’ll explain my situation:
I was given the task of putting up a graylog server for the security team to check for behavior anomalies in the administrative accounts. I’m not in the IT department, so I’m struggling a little, but I managed to do it.

The server

The server is installed in a VM with enough resources to make it function properly (it didn’t make me doubt that so far)

CentOS 7
graylog-server v3.1.0
elasticsearch-oss v6.6.2

What I need

We have a few hosts that send syslog messages to the Graylog server, something like a Domain Controller, a VPN, a couple of firewalls and some production server.
Unfotunately, it was the IT department that set up the input stream (just that) and it’s a single stream gathering every syslog message, so I will have to come out with some rule to divide by host because they are not going to change the input stream anytime soon.
I need, at this point of the task, a way to divide logs by whether they were generated during business hours or not.
Ideally, in the end I should be able to make an executive level monthly review with a couple of charts (just to stick to compliance rules, i guess)

The pipeline

I studied the documentation and did some research on google and on this forum (where the rule closest to my needs was in this topic) and i came with the following pipeline:

Stage 0 (All must be satisfied)
rule "has received time"
when
    has_field("EventReceivedTime")
then
end

rule "has target username"
when
    has_field("TargetUserName")
then
end

I was testing this pipeline on a limited set of results

Stage 1 (Any one can be satisfied)
rule "business hour flag"
when
    to_long(to_date($message.timestamp).hourOfDay) >= 7 &&
    to_long(to_date($message.timestamp).hourOfDay) <= 19 &&
    to_long(to_date($message.timestamp).dayOfWeek) >= 1 &&
    to_long(to_date($message.timestamp).dayOfWeek) < 6
then
    set_field("business_hour", "true");
end

rule "not business hour flag"
when
    to_long(to_date($message.timestamp).hourOfDay) < 7 &&
    to_long(to_date($message.timestamp).hourOfDay) > 19 &&
    to_long(to_date($message.timestamp).dayOfWeek) < 1 &&
    to_long(to_date($message.timestamp).dayOfWeek) > 5
then
    set_field("business_hour", "false");
end

Both rules were tested with and without quotes around the true/false values

The issue

Although i see some sort of throughput in both the stages of the pipeline, the throughput, which I expect it to be the same on the stages, is uneven. Moreover, and this is the real problem, in the search, none of the logs has the business_hour field

I set, as I saw in the documentation, this in the message processor configuration:

1	AWS Instance Name Lookup	active
2	GeoIP Resolver			active
3	Message Filter Chain		active
4	Pipeline Processor		active

Any idea of the reason why this pipeline doesn’t set the new field?

welcome @kaaroten

nice work so far!

You only have a little glitch in your rules that might become visible if I rewrite that:

when
    to_long(to_date($message.timestamp).hourOfDay) >= 7 AND
    to_long(to_date($message.timestamp).hourOfDay) <= 19 AND
    to_long(to_date($message.timestamp).dayOfWeek) >= 1 AND
    to_long(to_date($message.timestamp).dayOfWeek) < 6
then

That condition could never be positive and the field will never be created. But if you change that to something that can happen - this will work:

when
    ( to_long(to_date($message.timestamp).hourOfDay) >= 7 AND
    to_long(to_date($message.timestamp).hourOfDay) <= 19 ) OR 
    ( to_long(to_date($message.timestamp).dayOfWeek) >= 1 AND
    to_long(to_date($message.timestamp).dayOfWeek) < 6 ) 
then

Get it?

What is also something to keep in mind. the date of the timestamp is UTC so you need to make the times UTC Times. While the documentation give the option to work with timezones, it is actually not working ( https://github.com/Graylog2/graylog2-server/issues/6486 ) but should™ be fixed soonish. After that the following will work properly:

rule "off work hours"
when
   ( to_long(to_date($message.timestamp, "Asia/Manila").hourOfDay) >= 0 AND to_long(to_date($message.timestamp, "Asia/Manila").hourOfDay) <= 6 ) OR
   ( to_long(to_date($message.timestamp, "Asia/Manila").hourOfDay) >= 18 AND to_long(to_date($message.timestamp, "Asia/Manila").hourOfDay) <= 0 )
then
   set_field("trigger_workhours_off", true);
end

You should also not forgot about weekends - as most companies do not run 24/7

rule "off work weekend"
when
   // from Monday (1) to Sunday (7)
   to_long(to_date($message.timestamp, "Asia/Manila").dayOfWeek) == 7 OR
   to_long(to_date($message.timestamp, "Asia/Manila").dayOfWeek) == 6
then
   set_field("trigger_workhours_off", true);
end

regarding having " around true and false - having "true" will create a string that writes true into the field. While just using true will create a boolean what is significant smaller to store. You need to decide what you need in your case.

Hi, Thank You for your reply.
Apparently I failed to understand the use of pipelines (and i think I miss it a bit right now)

Anyway, they told me that I was trying too hard, because being able to export results as a CSV was a viable solution, too.
Now I use my pipeline as a decorator in searches, instead of expecting it to modify logs/messages by creating another field (is it the intended use, BTW?).

The solution

I played with the excess two hours of my timezone and I came up with a solution that is working as intended right now:

rule "business hour flag"
when
    to_long(to_date($message.timestamp).hourOfDay) >= 7 &&
    to_long(to_date($message.timestamp).hourOfDay) <= 15 &&
    to_long(to_date($message.timestamp).dayOfWeek) >= 1 &&
    to_long(to_date($message.timestamp).dayOfWeek) < 6
then
    set_field("business_hour", true);
end

Apparently, this works fine, glitchless, because I give it two ranges (both workhours and workdays) that should work together.

rule "not business hour flag"
when
    to_long(to_date($message.timestamp).hourOfDay) < 7 ||
    to_long(to_date($message.timestamp).hourOfDay) > 15 ||
    to_long(to_date($message.timestamp).dayOfWeek) < 1 ||
    to_long(to_date($message.timestamp).dayOfWeek) > 5
then
    set_field("business_hour", false);
end

Both rules are tested and working, so this might be a solution for my case. I will give it a few days more for testing, but I’m hoping.

What I would like to know right now is whether is possible to work with minutes this way or not. For example, in the case I wanted to consider business hours 8.30 through 18.30.
More specifically: is it possible to use parenthesis to group conditions/logical statements?

More specifically: is it possible to use parenthesis to group conditions/logical statements?

Did you checked the example rules I have given?

So, yes that is possible.