We have setup multinode graylog 3.0.2 cluster for our production logs processing. Daily logs size are 250 GB to 300 GB. We are using filebeat log shipper and graylog sidecar to upload logs.
Below are the setup architecture.
Graylog + MongoDB cluster
Three nodes with 128 GB RAM, 32 CPU, 700 GB Disk and 64 GB heap for each.
Three nodes with 128 GB RAM, 32 CPU, 700 GB Disk and 64 GB heap for each. Where two nodes are Master + Data and one is only Master. And 18 TB SAN mount for Data node.
We are uploading last day logs i.e. Yesterday’s logs.
We have split the logs in three graylog nodes. Copy these logs files to one location from where graylog sidecar will upload the logs using filebeat log collector.
- When logs uploading started from all nodes the input speed are 50k to 100K msgs/s and output speed is 20k to 100k msgs/s. Batch size set to 50k msgs
- After some time journal gets full and output logs stop processing.
Is there anything missing or anything wrong. Please suggest.