Ideal archiving method?

We are looking for ideas on how to config the enterprise archiving.
On one test server we are finding that it takes around 30 minutes to archive 10GB of events. On our production machine, we split off indices at 28GB. The production machine does have double the CPU cores, however on our test machine we haven’t seen much of a CPU hit during compression. Is there a way to increase the threads or is it possible to have archive compression performed on another machine?

We don’t want to run into a scenario where it can’t keep up with archive compression because it takes 2 to 3x longer to compress as compared to writing the index

This hints to disk I/O being the bottleneck and not CPU performance. You can speed up things by using faster disks e. g. SSDs instead of spinning disks, local storage instead of network storage, etc.

Makes sense, because in this case it is pulling the indices from a NAS through a 1Gbps connection and writing to that same NAS simultaneously while also writing new events over the same connection. I guess we could always have it write to local disk and then setup something to move the archives onto the NAS afterwards. It wouldn’t show what we have archived on the NAS though

I’m also interested in your solution Karit. We are also archiving around ~10-12Gb per index and are seeing archival times around 30-40 minutes. Were you able to determine a good solution for this and if so, what helped out the most in your architecture?

We are going a different route and are testing out elasticsearch snapshots instead of the archiving offering from graylog. Graylog’s solution is very simple to use and keeps things organized well, however it’s not robust enough to allow restoration of multiple archives simultaneously
Oddly enough, CPU and disk I/O stats are well within spec and hardly being utilized when we used graylog’s archiving/restore and there wasn’t much of a performance change when testing with local disk, so it’s still a bit of a mystery as to why the process takes so long. We don’t have the ability to test with SSDs at this time

@karlt If you have an enterprise support contract, please reach out to us so that we can work on a solution together or tune your setup to perform the archiving and restore procedures faster.