Pipeline - Tab Split

Graylog Central (peer support)
1

Hello,

I have some IDS logs coming into Graylog, I am trying to split the fields out to report using:

let m = split("\\|", to_string($message.message));
set_field("ts", m[0]);
set_field("uid", m[1]);

The logs fields are separated by a tab, whats the best way of getting these into associated fields. Can I do this in Graylog?

Cheers
Pete.

2

Do you have examples of these messages you could share?

3

Yep here is the the message just changes the IP for Googles:

1516195367.174058	CNDYE23uGhnLzFS9J8	8.8.8.8	23872	8.8.8.8	53	udp	54914	-	r4.res.office365.com	-	-	-	-	0	NOERROR	T	F	F	F	0	r4.res.office365.com.edgekey.net	300.000000	F

The split pipeline rule is as follows:

rule "Extract bro_dns log fields"
when
  has_field("application_name") &&
  contains(value: to_string($message.application_name), search: "bro_dns", ignore_case: true)
then
let m = split(",\t", to_string($message.message));
  set_field("ts", m[0]);
  set_field("uid", m[1]);
  set_field("id_orig_h", m[2]);
  set_field("id_orig_p", to_long(m[3]));
  set_field("id_resp_h", m[4]);
  set_field("id_resp_p", to_long(m[5]));
  set_field("proto", m[6]);
  set_field("trans_id", m[7]);
  set_field("query", m[8]);
  set_field("qclass", to_long(m[9]));
  set_field("qclass_name", m[10]);
  set_field("qtype", to_long(m[11]));
  set_field("qtype_name", m[12]);
  set_field("rcode", to_long(m[13]));
  set_field("rcode_name", m[14]);
  set_field("AA", m[15]);
  set_field("TC", m[16]);
  set_field("RD", m[17]);
  set_field("RA", m[18]);
  set_field("Z", to_long(m[19]));
  set_field("answers", m[20]);
  set_field("TTLs", m[21]);
  set_field("rejected", m[22]);


end
4

Since there is no information about the field name in your message whatsoever, it’s not possible to completely automatically extract that information into message fields.

But you could create a matching grok pattern if the structure of the message doesn’t change:

5

But I am trying to set the field name by splitting it? I think the issue is whether the split function can split “TAB” or not?

Does that make sense?

6

Yes, the split() function supports tabs, or really anything you can express in a regular expression.

See graylog-plugin-pipeline-processor/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/split.txt at 2.4.0 · Graylog2/graylog-plugin-pipeline-processor · GitHub for examples.

7

Think I am making progress but getting errors in server.log now for any new field:

" error=<{“type”:“mapper_parsing_exception”,“reason”:“failed to parse”,“caused_by”:{“type”:“illegal_argument_exception”,“reason”:“Can’t parse [index] value [not_analyzed] for field [id_orig_h], expected [true] or [false]”}}>"

Any ideas?

8

Try rotating the write-active index (System/Indices/Index Set/Maintenance).

9

Hello, Thanks for coming back Jochen.

I can’t see that, running Graylog 2.4. Any pointers?

10

Ignore me, found it.

11

All working, the rotate fixed it. Thanks for the help.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.