Aggregates Plugin for Graylog

the NEW Marketplace Plug-Ins
1

@cvtienhoven

Stargazers: https://github.com/cvtienhoven/graylog-plugin-aggregates/stargazers

Aggregates Plugin for Graylog

Build Status

Required Graylog version: 2.5.0 and later - NOT BACKWARDS COMPATIBLE WITH OLDER GRAYLOG VERSIONS .

Note : When upgrading from 1.x.x to version 2.0.0 of the plugin, it’s required to modify existing rules (regarding streams and alerting) and existing report schedules (regarding receivers). See the screenshots below.

The Aggregates Plugin for Graylog enables users to execute term searches and get notified when the given criteria are met. Currently, there are the following alert conditions in Graylog:

  • Message count
  • Field value
  • Field content

However, these conditions will not be sufficient to match the following scenario:

Send an alert when someone fails to login from the same source ip 3 or more times in the past minute.

This scenario is actually very useful in a security context, but with the built-in alert conditions, it’s not possible to match exactly this condition. The part " from the same source ip " is difficult to match. It would take an aggregate search that groups by value and returns the count per value. That’s what the plugin aims to do. You can configure rules as shown in the screenshots below.

Create / edit a rule

Generated Alert Condition

Alert Overview

Rule overview

Report schedule overview

Create / edit a report schedule

Report example

Define the Aggregates Email Callback

Email example

Usage

Permissions

Use the Aggregates tab in the web interface of Graylog to define rules with alert criteria. For non-admin users, there are the following permissions that should be configured (via the REST API) to be able to fully (or partly) operate the plugin:

  • aggregate_rules:create
  • aggregate_rules:read
  • aggregate_rules:update
  • aggregate_rules:delete
  • aggregate_report_schedules:create
  • aggregate_report_schedules:read
  • aggregate_report_schedules:update
  • aggregate_report_schedules:delete

Note : To be able to view the list of rules, non-admin users need both the aggregate_rules:read and the aggregate_report_schedules:read permissions.

Each rule can be configured to be executed on a particular stream. For the latter option to be present, the user needs to be able to have at least the following permissions:

  • searches:absolute
  • searches:relative
  • searches:keyword

Sending alerts

Since version 2.0.0, the plugin integrates tightly with the Notifications and Alert Conditions within Graylog. You can define a notification on a stream as you would normally do. The plugin creates an Alert Condition when creating an Aggregate Rule and it keeps the condition in sync with the rule after updates. In previous versions, you could only send emails, but now you can also use the HTTP Alarm Callback for instance.

New in version 2.3.0 is the Aggregates Email Alarm Callback . This callback works in the same way as the normal Email Alarm Callback, but a table (inserted at the ${matchedTermsTable} placeholder) is added to the template, as shown in the screenshot. The table contains the found values and their number of occurrences, with links to the respective search query. This callback sends an HTML email, so you can customize the layout by using HTML tags and CSS in the email template.

Note 1 : If you delete the Alert Condition, the plugin re-creates it, except when you disable the rule.
Note 2 : Enabling the message backlog can inflict a performance penalty, as the backlog has to be assembled from the found terms, using separate searches. Use with care.
Note 3 : Alert Conditions are created under the user admin .
Note 4 : For the Aggregates Email Alarm Callback, only email receivers can be defined, user receivers are not supported.

Reporting

In the rule overview, there’s an option (checkbox) to include rule history in a report. This report is a PDF file that contains a bar chart for every rule, summing up the total number of hits for that rule per period. The grid for the chart is automatically determined based on the total amount of time. The report is tailored per alert receiver, which means that a receipient will only receive charts for the rules subscribed to.

When creating or editing a rule, the schedule(s) for generating report(s) can be supplied. For configuring a schedule, you should supply a name, a valid Cron expression using the Drools syntax and the timespan, e.g. the amount of history you wish to incorporate in the report. Since version 2.0.0 of the plugin, receivers of reports are defined on the report schedule, not on the aggregates rule anymore.

Note : The maximum timespan determines the overall retention. So, if you have a report schedule that takes a year of history, the retention of hits will be a year. This might influence your MongoDB storage needs.

Changelog

2.4.0

  • Bugfix: Build against 2.5.x version (#46). This breaks compatibility with 2.4.x versions of Graylog.
  • Feature: Added the REST endpoints to the Audit Event Processor (#43).
  • Bugfix: Escape query terms when compiling the message backlog (#44).
  • Bugfix: Use correct descriptor for Alert Condition (#42).

2.3.0

  • Feature: Added the Aggregates Email Alarm Callback that emails a table with found field values, the # of occurences and a link to the search (#35, #41).
  • Bugfix: Removed the extra " AND streams: " from the query, as the stream is already filtered in the Alert Condition.
  • Bugfix: Logged the removal of history items on debug level instead of info (#26).
  • Bugfix: If the stream is altered for a rule, remove the AlertCondition on the original stream first.
  • Bugfix: Altered description for stream title in rule list if user can’t see that stream.
  • Bugfix: The Aggregates item in the navition bar is only visible when users have aggregate_rules:read permissions (#40)

2.2.4

  • Bugfix: Use Router.pluginRoute() to link to page in plugin (#34)
  • Bugfix: Resolve unresolved alerts whose duplicate (same timestamp) has already been resolved. Duplication needs further investigation.

2.2.3

  • Bugfix: Removed decoding of URI that was not needed (#32)
  • Bugfix: Corrected timestamp in returned CheckResult for alert generation (#33)

2.2.2

  • Build against Graylog version 2.4.x

2.2.1

  • Bugfix: fixed NullPointerException in AggregatesMaintenance when rule has no alertConditionId. Also log a warning when this is the case.

2.1.1

  • Bugfix: Creation of new rules failed with error 500
  • Bugfix: With only one available stream, corrupt rules got created
  • Bugfix: In the API browser, duplicate entries existed for Plugin/Aggregates (#21)

2.1.0

  • Fully integrated alerting with the native Graylog notifications method, including message backlog (#24, #25)
  • Removed the “sliding” parameter, because the generated AlertCondition is always evaluated every minute
  • Fixed bug that renaming a rule wasn’t reflected in history
  • Implemented functionality so no results would trigger an alert if theres a “less than” condition (#11)

2.0.0

  • Moved report subscriber option to the Report Schedules screen
  • Introduced alerting via the native Graylog Alarm Callbacks (Notifications)

Getting started

This project is using Maven 3 and requires Java 8 or higher.

  • Clone this repository.
  • Run mvn package to build a JAR file.
  • Optional: Run mvn jdeb:jdeb and mvn rpm:rpm to create a DEB and RPM package respectively.
  • Copy generated JAR file in target directory to your Graylog plugin directory.
  • Restart the Graylog.

Plugin Release

We are using the maven release plugin:

$ mvn release:prepare
[...]
$ mvn release:perform

This sets the version numbers, creates a tag and pushes to GitHub. Travis CI will build the release artifacts and upload to GitHub automatically.

Installation

Download the plugin and place the .jar file in your Graylog plugin directory. The plugin directory is the plugins/ folder relative from your graylog-server directory by default and can be configured in your graylog.conf file.

Restart graylog-server and you are done.

Development

You can improve your development experience for the web interface part of your plugin dramatically by making use of hot reloading. To do this, do the following:

  • git clone https://github.com/Graylog2/graylog2-server.git
  • cd graylog2-server/graylog2-web-interface
  • ln -s $YOURPLUGIN plugin/
  • npm install && npm start

Your Rating

Please sign in to rate this add-on.

Comments

nix-power almost 3 years ago

graylog 3.x compatibility ?

cvtienhoven over 3 years ago

For any issues, please head over to GitHub and file an issue when needed, as I don’t check the comments on the marketplace very often. Cheers!

safecyn almost 4 years ago

Is this plugin up to date with Graylog 2.4? For some reason when I installed it, the aggregates tab shows up, but not even the admin user has permissions to edit schedules, not even the pre-existing sample ones.

shchbo about 4 years ago

when i try to send an alert with out the option of “in report” it does not work. it sends the mail only when to check box is checked. graylog 2.3.0

mhocso over 4 years ago

Nice plugin! Is it possible to add the $field value to the alert message? So if source is used for tracnig failed logins for example can the alert contain the source that matched (ex 10.10.10.1) Right now it does alert but i need to fill the IP for callback to the website.

Kakuden over 4 years ago

@cvtienhoven I installed 2.1.1 and could create a aggregates rule.
Thanks!!

cvtienhoven over 4 years ago

@Kakuden: this was a bug, I released a fix for this today (version 2.1.1). Should be available on Marketplace soon, or else on the Github page: Releases · cvtienhoven/graylog-plugin-aggregates · GitHub

Kakuden over 4 years ago

I installed this plugin(2.1.0) on virtual appliance(2.3.1-3).
When I tried to create Aggregate Rule, the following error message appeared and failed to create rule.

Could not create rule
Creating rule failed with status: cannot PUT http://:9000/api/plugins/org.graylog.plugins.aggregates/rules (500)

Are there any ways to fix this?

elvioorg over 4 years ago

It’s a great plugin, but I’ve a problem to use with HTTP Alert Notification.
I have configured a notification with HTTP ALARM CALLBACK in aggregate plugin, but I get the HTTP POST messages empty as reported here below:

GET:Array
(
[function] => graylog
)
POST:Array
(
)
Does anyone know how to configure (custom) an http callback? THANK YOU
All email callbacks work perfectly.

cvtienhoven over 4 years ago

@Harris108:
In the query you can use AND, but it’s not possible (yet) to combine two fields. You’d have to introduce a field yourself in which you concatenate the two fields for instance, and use that new field in the plugin.

Harris108 over 4 years ago

Nice plugin. Just want to know if this plugin can be used to make a query of two fields together with an AND ?
for Example
Query : Root AND signin.amazonaws.com
Filed : user_type AND user_agent

Thanks

cannojr over 4 years ago

Excellent work on this plugin.

berekese over 4 years ago

Hi, I’m using Graylog v2.3.0-alpha.3+c795033 but when I put that plugin on his directory and I restart the graylog-server doesn’t start. I can’t see any logs error, this is last lines:
2017-06-08T09:37:30.212+02:00 INFO [CmdLineTool] Loaded plugin: Aggregates 1.0.1 [org.graylog.plugins.aggregates.AggregatesPlugin]
2017-06-08T09:37:30.214+02:00 INFO [CmdLineTool] Loaded plugin: Elastic Beats Input 2.3.0-alpha.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-06-08T09:37:30.215+02:00 INFO [CmdLineTool] Loaded plugin: Collector 2.3.0-alpha.3 [org.graylog.plugins.collector.CollectorPlugin]
2017-06-08T09:37:30.216+02:00 INFO [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.3.0-alpha.3 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-06-08T09:37:30.216+02:00 INFO [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.3.0-alpha.3 [org.graylog.plugins.map.MapWidgetPlugin]
2017-06-08T09:37:30.217+02:00 INFO [CmdLineTool] Loaded plugin: NetFlowPlugin 0.1.0 [org.graylog.plugins.netflow.NetFlowPlugin]
2017-06-08T09:37:30.224+02:00 INFO [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.3.0-alpha.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-06-08T09:37:30.225+02:00 INFO [CmdLineTool] Loaded plugin: Anonymous Usage Statistics 2.3.0-alpha.3 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-06-08T09:37:30.225+02:00 INFO [CmdLineTool] Loaded plugin: SnmpPlugin 0.3.0 [org.graylog.snmp.SnmpPlugin]
2017-06-08T09:37:30.444+02:00 INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackT
raceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=deb
2017-06-08T09:37:30.646+02:00 INFO [Version] HV000001: Hibernate Validator null

Any idea? Thanks.

vsegdacocacola over 4 years ago

Great job! +1 to have alerts in Graylog itself

cvtienhoven almost 5 years ago

@BlackPearl01: Scheduling of reports is now included (as of version 1.0.0).

BlackPearl01 almost 5 years ago

Hello,

First, thank you for your plugin for GrayLog! It’s useful and help me to create some alerts in GrayLog.

The PDF report are very useful too! But as you had write, for the moment, the configuration of intervals, schedules etc. for reports is not possible yet.

Have you an idea of the date when you can develop and include this feature in your plugin?

Thank you again,

Nemesis741 over 5 years ago

Nice Plugin. It would be cool if the alarms of these agregates could be counted and added to a dashboard.

We sometimes have duplicated requests on some of our webservers, so it would be nice to add a counter like “There were 15 duplicates in the last hour” to our dashboards.

aarvee11 over 5 years ago

Loved the plugin. Awesome work bro!

Please sign in to comment.