Unable to extract data from flowlogs and send as CSV file

I am having trouble in configuring Graylog notification. I am feeding graylog my flow logs via kinesis as input. In those flowlogs, I have rejected IPs as “2 ID someID IP destination_ip port 28679 6 19 6843 1653903019 1653903048 ACCEPT OK” I want to set up email notification and I have already set email parameters in .conf file.

My environment:

  • Installed graylog:4.2.3-1 cluster using Helm.

  • all pods are working and healthy.
    elasticsearch-master-0 1/1 Running 0 25h
    graylogs-0 1/1 Running 0 25h
    graylogs-mongodb-0 1/1 Running 0 25h
    graylogs-mongodb-arbiter-0 1/1 Running 0 25h

**I tried to set alerts for those IPs but if there are 1000 results, I would be getting 1000 emails without attaching those IPs in email. **

I don’t know what to do next like how can I recieve the csv file daily for rejected IPs in extracted from my flow logs input. sorry if my explanation is a bit untidy

Hello && Welcome @hamzasaeed5594

Couple ways you can archive this.

  1. You can get the Enterprise license which is free under 2GB, then you can use reports and/or the Event Correlation.
  2. Filter out the messages needed into a stream and attach the Event Definition to that stream.
  3. Adjust your grace period on the notification to 23-24 hours. maybe fine tune the Search Query.
  4. Under Event Definition the " Aggregation" works good.

A better understanding can be found here && here.

1 Like

Thanks a lot taking your precious time out for answering my query. I am sending this message because I can’t install enterprise free 30 days trial.

Also for the clarification, I will be attaching the csv file of all rejected IPs in last 24 hours to an email. This emails should be sent every day along with attached csv file in automated way.

Hello,

You can manipulate the Notification template to show the IP address like I stated above.
Example:

Fields:
${foreach event.fields field}  ${field.key}: ${field.value}
${end}
${if backlog}
--- [Backlog] ------------------------------------
Last messages accounting for this alert:
${foreach backlog message}
TargetUserName:    ${message.fields.TargetUserName}
WorkstationName:   ${message.fields.WorkstationName}
EventReceivedTime: ${message.fields.EventReceivedTime}
Source:            ${message.source}
${end}

This would depend on how this Alert/Event Definition is setup.
Create a stream then some rule specifying what messages/logs needs to go in there. This is done by rules defined in that stream. The create the Event Definition and attach it to that stream.
The links I post above should help.

Not sure what going on with that error show about enterprise license and how you went about requiring one. Unless you are above 2 GB?

first of all I am getting this error using the body template that you provided

Secondly I tried with the default body template in notifications (email) but didn’t get the message or ip in the mail.
image

I have set the stream then pipeline rules as you have told me to do.
Here are the event definition settings:



I am not sure what is wrong here.

Hello,

Well that was an example of what you could do, I was not expecting a Copy & Paste, perhaps I should have mentioned that. That error means you needed a ${end} at the template.

I’m going to demonstration the Alerts for you, perhaps this will help.

This will be an Example but your environment is probably different then mine. Keep an eye on tic boxes, etc…

In this example Im going to get the host_names from those messages by count and send a notification alert in 24 hours with 5 message backlogs.

1.I created a stream called Beats Filter I also removed it from “All message” In my case to save space.

image

2.Created event definition using the filter, Notice the quotes && the filter Preview? Its to insure I have correct configurations made.

3.I want to be alerted if more then 1 messages hits this stream that has “The Network Setup Service service entered the stopped state” within those messages. Which would be greater the “0” meaning if one messages arrives, Alert.

4.Notification template, what I need to get is the fields that has an IP address or Hostname. As shown below I’m going to use host_name field

5.Create my custom template. For this demonstration called " Beat Filter". I will be remove the HTML section of my Email notification template and just have the configurations shown.

--- [Event Definition] ---------------------------
Title:       ${event_definition_title}
Description: ${event_definition_description}
Type:        ${event_definition_type}
--- [Event] --------------------------------------
Timestamp:            ${event.timestamp}
Message:              ${event.message}
Source:               ${event.source}
Key:                  ${event.key}
Priority:             ${event.priority}
Alert:                ${event.alert}
Timestamp Processing: ${event.timestamp}
Timerange Start:      ${event.timerange_start}
Timerange End:        ${event.timerange_end}
Fields:
${foreach event.fields field}  ${field.key}: ${field.value}
${end}
${if backlog}
--- [Backlog] ------------------------------------
Last messages accounting for this alert:
${foreach backlog message}
hamzasaeed5594 was here:  ${message.fields.host_name} <<<---- Here is my new configuration.
${end}
${end}

6.Back to my Event definition by adding the new Notification template. Noticed the Grace Period and backlogs section?

7.Email Results

Conclusion:

I creates a stream to fillter out specific logs, then Create a search filter in the event definition to filter even further and added a “COUNT” greater then “0” meaning more them one of those messages.
On the notification side I have a grace period for 24 hour with 5 backlogs for this example.
Configured my Notification template to only show “host_name” and from the last picture it also shows the count on how many message I have.

hope that helps

Thanks working fine now :slight_smile:

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.