Seems a bit low for that many nodes but I have found that Grok patterns can have a huge impact if not done correctly. We do use some but have learned it is MUCH faster and performance gaining to send the logs preformatted in GELF from the source. Not possible in all instances but a huge benefit where the option is available. NXLOG does a great job of this when configured for it. In general we process around 10-15k per second on just two nodes of around 4 cpus and 16 GB memory right now. Even with those specs We only generally have those nodes hit around 50% utilization most of the time.
Mantil
(Matt)
5
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Graylog not processing messages with Grok pattern | 0 | 662 | March 23, 2017 | |
| One complex grok Vs a set of regex extractors | 5 | 2291 | April 20, 2020 | |
| Processbufferprocessor threads consuming very high CPU load | 3 | 1229 | November 6, 2017 | |
| Process buffer gets full with the Grok pattern Extractor | 2 | 1326 | April 9, 2019 | |
| Journal growth too much when I enable an extractor | 9 | 647 | March 18, 2021 |