Math between fields


#1

I extracted 2x fields - size and time. Now I need to divide one on another and alert the result if threshold is reached. Could you please advice how to do math between fields?


(Jochen) #2

Graylog currently doesn’t support running ad-hoc mutations of indexed data, but you can use processing pipeline rules to run calculations on messages and store the result into new fields, see http://docs.graylog.org/en/2.2/pages/pipelines.html for details.


#3

Thank you for your advice, jochen.
How can I create a rule to divide one field on another. I didn’t find that in docs. Described functions don’t do that.

Thank you !


(Jochen) #4

You can use basic arithmetic operations as demonstrated in this test case:

Make sure to cast the fields to the proper type (e. g. with to_double()) before doing any arithmetics.


#5

So I created the pipeline rule:

rule "WRITE bandwidth"
when
  has_field("totaltime") && has_field("sentbytes")
then
    let time = to_double($message.totaltime);
    let size = to_double($message.sentbytes);
    let bandwidth = size / time;
    set_field("wrbandwidth", bandwidth);
    route_to_stream(name: "All WRITE stream");
end

wrbandwidth field has be be created now. Did I catch you explanation correctly?

How to use new field to create a graph in a dashboard?

Sorry for the newbie questions. I’m using graylog for few hours but need to do the tasks ASAP.
Thanks for your support !


(Jochen) #6

Yes.

Just like any other numeric field: http://docs.graylog.org/en/2.2/pages/dashboards.html#examples


#7

I created the rule described above but the new field wrbandwidth appeared only after a while.
Next I created couple of more pipelines with new fields calculated by rules (read_bandwidth and writhe_bandwidth). After half an hour new fields are still not visible.

How long it takes to get new field visible in a search?
Can we speed up that?


#8

Finally both new fields appeared. But the questions are still the same.

How long it takes to get new field visible in a search?
Can we speed up that?


(Jochen) #9

This depends on your setup and a lot of parameters, but usually it’s a few seconds.


(Jochen) #10

Only new messages, which run through the processing pipeline, will have the new fields.


#11

I got the idea, thanks.

Everything works just fine now.
I really love Graylog !

Jochen, thank you for your support !!!


(Mohamed) #12

Hello all,

I’m replying to this old post as it seams doesn’t work for me. I created a pipeline rule to calculate the upload speed based on extracted fields from IIS logs (cs_bytes & response_time_msec).

rule "WRITE UploadSpeed"
when
  has_field("cs_bytes") && has_field("response_time_msec")
then
    let time = to_double($message.response_time_msec)/1000.00;
    let size = to_double($message.cs_bytes);
    let bandwidth = size / time;
    set_field("UploadSpeed", bandwidth);
    route_to_stream(name: "Web logs");
end

My problem is that the the criteria never come back as true. I used the “Web Logs” stream as the Pipeline connection to be able to use the extracted fields, but this didn’t help.

I wonder is this is something was suitable for older Graylog version (I’m using Graylog v2.4) or it is something I’m doing wrong?

Thanks in advance…
Mohamed


(Jochen) #13

Do these fields exist when the pipeline rules are running? Are they created by some extractors?

Check the order of message processors in your Graylog cluster on the System/Configurations page.


(Mohamed) #14

Hi @jochen

My Message Processors Configuration was in the following order.

# Processor Status
1 AWS Instance Name Lookup active
2 GeoIP Resolver active
3 Pipeline Processor active
4 Message Filter Chain active

I pushed the Pipeline processor to be #4, and will try again

Thanks you.
Mohamed


(Mohamed) #15

Hi @jochen

It works :slight_smile: Thank you for the tip.

Thank you
Mohamed