I have maybe a stupid question: how to monitor (and get notifications about) the Graylog server itself?
I have setup various alerts and notifications for the logs I send to Graylog, and it works very good. The problem is when Graylog itself has a problem, you just stop receiving any notifications from it.
For example, the most common error I have is when Elasticsearch service crushes. When it happens, Graylog stops generating any alerts (as it cannot perform search in the index). When I go to Graylog web interface, I see the error, and I can restart Elasticsearch service. But I would like to have a notification about the failure (e.g., via email).
What would be the best way to monitor the status of Graylog itself? It’s kind of “who shaves the barber” issue: Graylog is supposed to notify about the problems with all your services (which supposedly send all logs to Graylog), but who notifies about the errors with Graylog?
Thanks in advance