Not Receiving Alerts from Graylog for Defined Events

Graylog Central (peer support)
1

Hello,

We have the dashboard setup in Graylog called Mishawum:

We had created a couple of events and set up notifications for our servers both Mishawum/ Revere to send alerts when an error count reaches a particular threshold but we wouldn’t receiving alerts in an expected manner. It seems like there is some issue with the Graylog application.

Events:

  • Test Alert - Mishawum Error count hits threshold
  • Test Alert - Revere Error count hits threshold

Notification:

  • Mishawum/ Revere Error Alert

And also observed that getting alerts for mishawum sometimes and that is also not at the exact time when an event rule is matched but after a long time (day after, 2 days after, etc…). As per my understanding, we may or may not receive alerts even though our message passed through the event rule/s.

For example, we defined an event to send alerts when the mishawum error count reaches 30. Getting alerts a day later or whenever Graylog sends by saying that the count is in the range of some 30-38 (but when looking at the dashboard actual count is somewhere between 400-450) after that not receiving any alerts even though there is a delay in the process.

Due to this, we couldn’t able to identify when our servers are down or having some issues.

Please let me know what’s causing the issue here? and also I’m seeing a lot of open incidents about the Graylog alert system.

2

Hello && welcome

This sound like a date/time issue but not 100% sure. To check, navigate to System --> Overview

Should loook like this:

image

Also check your Graylog configuration file make sure the time zone is correct.

root_timezone = America/Chicago

Check the date/time on Graylog server also.

timedatectl

If the date/times are correct then I need to ask a couple questions.

  1. Can you show your Alerts/Event Definitions configurations?
  2. What version of Graylog do you have?
  3. Did this always happen or did it just start?
  4. If this just started was there any updates applied prior?
  5. The sources sending message to Graylog are the date/time correct ?
  6. The log shippers on these devices, do you see any errors/warnings in the log files?
3

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.