Pipeline - Tab Split

(Peter Griggs) #1

Hello,

I have some IDS logs coming into Graylog, I am trying to split the fields out to report using:

let m = split("\\|", to_string($message.message));
set_field("ts", m[0]);
set_field("uid", m[1]);

The logs fields are separated by a tab, whats the best way of getting these into associated fields. Can I do this in Graylog?

Cheers
Pete.

(Jochen) #2

Do you have examples of these messages you could share?

(Peter Griggs) #3

Yep here is the the message just changes the IP for Googles:

1516195367.174058	CNDYE23uGhnLzFS9J8	8.8.8.8	23872	8.8.8.8	53	udp	54914	-	r4.res.office365.com	-	-	-	-	0	NOERROR	T	F	F	F	0	r4.res.office365.com.edgekey.net	300.000000	F

The split pipeline rule is as follows:

rule "Extract bro_dns log fields"
when
  has_field("application_name") &&
  contains(value: to_string($message.application_name), search: "bro_dns", ignore_case: true)
then
let m = split(",\t", to_string($message.message));
  set_field("ts", m[0]);
  set_field("uid", m[1]);
  set_field("id_orig_h", m[2]);
  set_field("id_orig_p", to_long(m[3]));
  set_field("id_resp_h", m[4]);
  set_field("id_resp_p", to_long(m[5]));
  set_field("proto", m[6]);
  set_field("trans_id", m[7]);
  set_field("query", m[8]);
  set_field("qclass", to_long(m[9]));
  set_field("qclass_name", m[10]);
  set_field("qtype", to_long(m[11]));
  set_field("qtype_name", m[12]);
  set_field("rcode", to_long(m[13]));
  set_field("rcode_name", m[14]);
  set_field("AA", m[15]);
  set_field("TC", m[16]);
  set_field("RD", m[17]);
  set_field("RA", m[18]);
  set_field("Z", to_long(m[19]));
  set_field("answers", m[20]);
  set_field("TTLs", m[21]);
  set_field("rejected", m[22]);


end
(Jochen) #4

Since there is no information about the field name in your message whatsoever, it’s not possible to completely automatically extract that information into message fields.

But you could create a matching grok pattern if the structure of the message doesn’t change:

(Peter Griggs) #5

But I am trying to set the field name by splitting it? I think the issue is whether the split function can split “TAB” or not?

Does that make sense?

(Jochen) #6

Yes, the split() function supports tabs, or really anything you can express in a regular expression.

See https://github.com/Graylog2/graylog-plugin-pipeline-processor/blob/2.4.0/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/split.txt for examples.

(Peter Griggs) #7

Think I am making progress but getting errors in server.log now for any new field:

" error=<{“type”:“mapper_parsing_exception”,“reason”:“failed to parse”,“caused_by”:{“type”:“illegal_argument_exception”,“reason”:“Can’t parse [index] value [not_analyzed] for field [id_orig_h], expected [true] or [false]”}}>"

Any ideas?

(Jochen) #8

Try rotating the write-active index (System/Indices/Index Set/Maintenance).

(Peter Griggs) #9

Hello, Thanks for coming back Jochen.

I can’t see that, running Graylog 2.4. Any pointers?

(Peter Griggs) #10

Ignore me, found it.

(Peter Griggs) #11

All working, the rotate fixed it. Thanks for the help.

(system) closed #12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.