Completely cloned our environment to Dev and isolated the environment to it’s own closed network.
First thing I tried is set all the indexes to only keep one and rotated them to clear out almost all data except for the data from the closed environment feeding in still. Didn’t see much difference in performance.
Rolled back the changes and tried downgrading mongodb. Didn’t notice any substantial difference. I didn’t notice mongod spiking on page load but that might be because the dev system is relatively idle.
Disabled tls, nginx, haproxy and loaded right from the graylog http page. No difference.
Tried reinstalling the Graylog rpm, no change.
I’m not seeing anything in the logs, the buffers are always empty, not sure what else to look for.
It seems like the graph processing on the search and sources page may be the culprit, but I could be off base.
Hoping @Jan, @jochen, or one of the other awesome people here have some insight on what to do next.
Edit: I believe it’s something to do with the calls being made on those pages, possibly something to do with the 4096 regression in the prior 3.0 release and how it was fixed. I tried rolling back to that release but it had the 4096 errors. I noticed on the current release when I search in a stream everything loads quick including the graphs.
