Yup, that’s what I was looking at doing…
- Normal Win/Lin boxen get the Sidecar Collector configured to use Beats.
- Syslog boxen would send their logs to a local collector which would forward through Beats…
Aduh… I just realised after living in Indonesia for 8 years and coming back to Europe I completely forgot my Indonesian. Dang.
For receivers going bye-bye I’d suggest the same as @jan - it’ll buffer nicely, just have to keep an eye on queue sizes so you don’t spend too long processing off the backlog if it happens 
1 Like
I added new data to my post.
I get more logs.
1 Like
More data servers soon! 
1 Like
I’ll add that we now have 3 graylog servers, 24 core beasts that sit at about 15% CPU usage due to our rather heavy use of pipelines; we also folded our “old” ES cluster into the Graylog one, so we’re now at 16 data nodes. We run indexes size-capped at 100Gb with 5 shards (for that sweet 20Gb shard size), and 3 replicas for availability and search speed.
Also went up to about ~2000 msg/sec on average, with peaks up to 7k/sec.
1 Like
The way you handle this is the one that will save your butt when the shit hits the fan - because what-ever you ingest can become multiplied by ongoing DOS-Attack, a misconfiguration by someone or something we can’t remember.
It is a good practice to not run your system on 90% but keep it under 50% load if you relay on the ingested logs - specially in worst-case situations.
2 Likes
I updated my post, we recognized the bigger GL heap is better somehow in our system.
2 Likes
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
Updated my post to reflect our current setup.
1 Like
I prepare a post too …actually 300GB per day 
2 Likes
Details! Details! 
1 Like
Is it really huge? 2.5Tb per day