A bit of background: We’re a small-ish ISP/MSP specializing in the education sector. We use Graylog to aggregate statistics from a wide variety of our services, such as email spam filtering, security tools, web hosting, etc. We seem to be pushing Graylog to its limits. We’re sending it about 5k messages per second, which it is injesting fine, but I have to be very careful if I run a search query, as it can cause the system to tip over. Injesting hangs, it is receiving the messages fine, but it can’t seem to index them, as outgoing messages goes to 0 per second. Sometimes it will recover on its own, in time, sometimes I have to restart the server, and hope it is able to come back up before my journal is filled.
But this is not a question about performance. After much fiddling and research and reading and experimentation, I have somewhat come to terms with the idea that during periods of high activity, I can’t expect to reliably run searches (though if anyone has any ideas or suggestion, I am open to hearing them). The problem is that we originally wanted to open up the graylog dashboard to our clients, to let them see, in real-time, how their services were doing, and what sort of value we were providing them. Suffice it say, that’s not really an option given that I can’t trust graylog to stay up even when it is just me verrrry caaaarrrefully running searches or loading dashboards - start letting clients run searches or open dashboards all willy-nilly, and it is sure to die.
What I’m wondering is if there is a way to export dashboards statically, or have them fronted by another service. The idea is that during periods of low activity, like at night, we run the necessary seraches to produce the dashboards for yesterday’s statistics, which we can then present to clients. We give up the ability to do it real-time, but I think we can live without that.
Thanks in advance.