I’d never heard of Graylog until a couple days ago. Love the idea but I can’t get it off the floor. I’m wanting to use it to monitor our ISP gear. syslog and netflow traffic. I see netflow errors in server.log, Error parsing NetFlow packet… but I don’t really think that’s causing this. Netflow’s are coming off a router running about 150 Mbps of traffic steady, and I have syslog setup on a handful of routers.
I followed the basic install for Ubuntu (16.04), and it starts up and responds okay but in a couple minutes it slugs to a halt. It’s like there is some memory or cpu leak but I can’t see where there is either, neither do I suspect that really because this looks like a stable package.
The backend becomes totally unresponsive to the web api calls and when I try to shut it down systemctl stop graylog-server.service takes 5 minutes or more to complete.
I did a complete reinstall to see if I borked something but same exact results.
It’s running on a single KVM guest, 4 cores, 10 Gigs of RAM. No ssl at this point, I wanted to get it stabilized first.
I’m really excited about what I see here but I can’t get it to run long enough to even troubleshoot. If someone can point me in the general direction I’d be grateful and I’m sure I can figure it out eventually.
Update: I blocked netflow traffic with ufw to see if it was that, and the graylog-server started responding a lot better. CPU went from 100+% (1 core) down to ~10%.
from the limited information you are giving, it looks like you kill that installation with more traffic it could handle. You need to configure your setup to be able to handle the known traffic you are sending over to it.
You should not overwhelm the setup, send only some logs. Learn what your setup can handle and tune your setup then add more and more traffic. It is like learning to drive a car, you won’t start with a fuel truck on your first hour.
Thanks for your response, is there an estimation on the time frame of the next release?
I found Graylog looking for a syslog solution but when I saw it did Netflow also, I committed immediately. If it’s not going to do Netflow for some time or won’t handle the amount of Netflow traffic we generate then I might just use it for syslog messages.
Right now, like I mentioned, I’m sending Netflows from one router that is running around 150 Mbps of traffic. I wanted to see that working and then I was going to start sending Netflows from a half dozen other routers. None of them are that busy, but together they would probably equal another 200 Mbps of traffic. Is this out of the scope of the Netflow plugin in Graylog?
@trendal I’ve been running the NetFlow plug-in for a year on stock OVA and it feeds from Sup 8E on a large multi-10gig SFP+ ports catalyst and I have no performance complaints. Visualizations can be better; etc but for a free tool it’s awesome!
Okay, I really didn’t see that this small amount of traffic would be overwhelming the application. If so then I’ll just quietly walk away to something else. I’ve had instabilities even after I removed all the Netflow traffic. The front-end usually hangs after using it for a couple of minutes. There are a ton of errors in the Javascript console, looks like framework errors griping about the backend going away to me, although I haven’t taken the time to look into it, but it finally just tosses up it’s hands and does nothing. Even an F5 refresh just generates a white page with the graylog header bar only. Eventually a reload works and it comes around for a while. It did get much much better after stopping the Netflow though.
But like you said, I’m willing to do a lot for a free tool. I’m blissfully ignorant on anything Java, but otherwise I’ll debug what I can, at least after I get my current project past release.
@trendal IMHO deploy the OVA and I bet you will see a totally different picture; installing the package, well whenever someone publishes an OVA there’s a darn good reason for it you get my drift… but seriously my exposure to Graylog is limited to the OVA and I’ve no plans to ever attempt a package install
you get my drift… but seriously my exposure to Graylog is limited to the OVA and I’ve no plans to ever attempt a package install