Assistance with extractors

Graylog Central (peer support)
1

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:

Trying to create extractors

2. Describe your environment:

  • OS Information:
    Debian Bullseye

  • Package Version:

  • Service logs, configurations, and environment variables:
    Not Appicable

3. What steps have you already taken to try and solve the problem?

Tried to create various regex’s, and use available Groks, but have I have failed.
I can get the first ip address, but I am stumped on the second ip address and the ports.

4. How can the community help?

Hi

I am a newbie using Graylog, and would welcome a bit of assistance in extracting fields from the following messages

A.
<129>Jun 13 06:34:34 DrayTek: [DOS][Block][Blocking List][45.143.200.50->255.255.255.255]

or

DrayTek: [DOS][Block][Blocking List][45.143.200.50->255.255.255.255]

The two IP address’s to be stored as new fields hackIp, ispIp for further analysis

B.

<129>Jun 11 14:27:12 DrayTek: [DOS][Block][tcp_flag, scanner=non_syn_ack_rst][2.57.122.225:3197->255.255.255.255:80][TCP][HLen=20, TLen=89, Flag=P, Seq=3197, Ack=0, Win=65535]

The two ip address’s and the associated ports

I am using the wizard to try to create the extractor on the input from the draytek, and place the extracted data into new fields

hackIp, hackPort, ispIp ispPort for further analysis

Thanks

Aimee

PS 255.255.255.255 is not my isp address :slight_smile:

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

2

Hello Aimee,

Click on the Message > Create extractor then Select extractor Type : Grok Pattern

Then Paste :

%{DATA:UNWANTED}\[%{IPV4:hackIp}\-\>%{IPV4:ispIp}\]

Do the same thing with

Grok pattern for this:

 %{DATA:UNWANTED}\[%{IPV4:hackIp}\:%{NUMBER:hackPort}\-\>%{IPV4:ispIp}\:%{NUMBER:ispPort}\]

it`s important to select “Named Captures Only”

Example:

community_Help1
community_Help1908×799 49 KB

2 Likes
3

Hi
Thanks for the assistance
Aimee

4

Hi

Since updating to v4.3.2+313b6bc. this grok
%{DATA:UNWANTED}[%{IPV4:hackIp}->%{IPV4:ispIp}]
and the other one you presented fail to work when tested. The input data format has not changed.
Any ideas why this has occurred?
Attached is a screen shot.

Kind Regards

Aimee

Screenshot 2022-06-25 at 11.28.10
Screenshot 2022-06-25 at 11.28.102363×932 194 KB

5

Hello AimDev,

you have to verify if the fields are still getting extracted, if they are then the extractor is still working.

about the error message, please be sure to load the right message (load another message)
load_another_message

for this search for: “DOS” AND “Blocking List” then click on the message you will see the message-id and and the index

6

Hi

Thanks for your help, it did work when I eventually got a block message.

Is it possible to extract the words Pass or Block, ip addresses, port, protocol from both messages, into the following fields

mode
src_ip
src_port
dst_ip
dst_port
protocol

Unfortunately Draytek is limited with its syslog output, it is not standard, can only supply one connection, there is no way to
filter the output.

Effectively I need to split the input, pass the relevant output to two separate streams for subsequent processing.

Many Thanks

Aimee

DrayTek: [FILTER][Pass][LAN/RT/VPN->WAN, 69:59:15 ][@S:R=13:1, 192.168.0.64:35164->255.255.255.255:53][TCP][HLen=20, TLen=52, Flag=AR, Seq=98691062, Ack=254576202, Win=0]

DrayTek: [DOS][Block][tcp_flag, scanner=urg_wo_ack][211.142.106.146:32450->255.255.255.255:4069][TCP][HLen=20, TLen=60, Flag=US, Seq=2319683347, Ack=0, Win=0]

7

hey @AimDev

I pipeline would work good for something like that, Just an Idea.

1 Like
8

Hi Folks,

as @gsmith suggested, create a pipeline and a rule for it, if you know how.
else you can create an Extrator type Grok pattern if you have a running System with enough Power !

For both Messages you can use:

%{DATA:UWANTED}\]\[%{WORD:mode}\]%{DATA:UNWANTED}%{IPV4:src_ip}:%{INT:src_port}->%{IPV4:dst_ip}:%{INT:dst_port}\]\[%{URIPROTO:protocol}\]

Just learn how to use Pipelines because they use less ressources, and route your Messages to Streams !

1 Like
9

Hi

Thanks again.
Actually I have got a number of pipelines working, however I had not realised I could use the filtering within a rule, and that it is more efficient. More homework!!

Aimee

1 Like
10

Just an fyi, My last lesson on pipelines was from here.

Something like this will work

rule "Regex multiple fields"
when
  has_field("message")
then

  let robin = regex("mode=(\\S+).*src_ip=(\\S+).*src_port=(\\S+)", to_string($message.message));

  set_field("mode",      robin["1"]);
  set_field("src_ip",    robin["2"]);
  set_field("src_port",  robin["3"]);  

end
2 Likes
11

@gsmith here is also an example
Using the Grok Pattern in a Pipeline !

rule "Our Rule example"
when
     has_field("message")
then 
     let fields_in_message = to_string($message.message);
     let var_our_grok_pattern = grok (pattern:"%{DATA:UWANTED}\\]\\[%{WORD:mode}\\]%{DATA:UNWANTED}%{IPV4:src_ip}:%{INT:src_port}->%{IPV4:dst_ip}:%{INT:dst_port}\\]\\[%{URIPROTO:protocol}\\]", value: fields_in_message, only_named_captures: true );
    set_fields (var_our_grok_pattern);
end
1 Like
12

Interesting.
Currently I split the input into two different streams, pipelines and indexes, vigor_block, vigor_allow.
This is done via a stream rule, mode = block or pass.
The mode is provided by the extractor on the syslog input from the vigor router.
Can a pipeline split into two separate streams with their pipelines, and indexes?.

13

When you create a Pipeline Rule you can then connect it to the before created Streams or to all messages

14

yes - you can use the route_to_stream() function. It’ important to note the message will finish the current stream before starting in the stream/pipeline it is routed to. If the stream the message is routed to has a different index at the end of it, the message would be stored in both indices.

2 Likes
15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.