Extract information out of message field

[hamzasaeed5594]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

Hi! I am trying to extract information coming from aws flowlogs in message filed as:

${srcaddr}    ${flow-direction} ${action} ${traffic-path} ${dstaddr}   ${region}  ${packets} ${protocol}
10.0.0.0          ingress         ACCEPT       -           10.0.0.1     us-east-1       23         6

in order to extract the ${srcaddr} ${flow-direction} ${action} ${traffic-path} from the whole message what can I do? What rule can be made please share an example to extract the above fields only and drop rest of the information.
the message field after filtration will look like this:
10.0.0.0 ingress ACCEPT -

[gsmith]

Hello,

@hamzasaeed5594

10.0.0.0 ingress ACCEPT - 10.0.0.1 us-east-1 23 6

I have create extractors for these, either use regex or GROK

DestinationIP Regular expression
extract data from message into dstip.
    Configuration
    regex_value: dstip=+?((?:\d+\.){3}\d+).+

Resluts:

Grok

grok_pattern: %{IP}

[hamzasaeed5594]

Thanks for the update @gsmith
I am sorry I think I didn’t clarify the requirements. You are trying to add extractor based on only 1 parameter which is destination Ip but instead I want to extract few inputs (like ${srcaddr} ${flow-direction} ${action} ${traffic-path}) from the message filed

and store it in the message field and drop the rest of the fields that I am not interested in, ideal message filed after extraction should look like this:

I was trying to play with extractors and came across with this:

turns out I can’t store my extracted data into message field. Can you please let me know if there is any workaround?

[hamzasaeed5594]

Also by calling an api from 1 day time stamp of message field, is it possible to have it extracted via api call?

[Aksel]

Hi Folks,

you could also create a Crok Pattern for it, click on the message > Create extractor then choose Grok Pattern:

Select Named Captures Only

Example : 192.168.1.1 ingress REJECT - 10.10.10.2 us-east-1 1 6

%{IPV4:srcaddr} %{WORD:flow-direction} %{WORD:Action} \- %{IPV4:destip} %{GREEDYDATA:location}

[hamzasaeed5594]

@Aksel when I click “try against example” I am getting this against your example,

[Aksel]

Please use the line i sent to you, and load the same message as you posted before

[hamzasaeed5594]

I used the same line as yours and the example message is also correct.

[hamzasaeed5594]

Sorry. It is working as I had new line space at the bottom that was hindering the execution.

[hamzasaeed5594]

can you please let me know where the extracted message will be going? How can I view them?

[Aksel]

When new same like messages comes in, the Fields will be extracted and these Fields will be shown before or after the message field: srcaddr, flow-direction, action etc …

see this example:

you can also use the Graylog search engine and search for: srcaddr: * AND Action: “REJECT” AND flow_direction: “ingress”

Btw go back to the created grok pattern and change flow-direction to flow_direction and save

[hamzasaeed5594]

thanks a lot! btw which api can I use to extract these fields from last 24hours timestamp?
Please note that I have tried /api/messages/index/messageID but it retrieves only one result. I need to have results for last 24 hours