- I have 3 clusters of Kubernetes (Dev, QA, Prod) that send their logs to Kafka with the logdriver gelf;
- The Kafka + Zookeper is clustered;
- I have 1 kafka topic for each Cluster Kubernetes (topic-DEV, Topic, QA, Topic-PROD);
My graylog connect in the zookepers and collects the logs according to my topic informed in my input.
- My Graylog is clustered: 3 nodes
- For Graylog I have a clustered mongoDB of 3 nodes;
- I have an Elasticsearch with 5 nodes for the graylog;
- I have 3 Inputs (input-DEV, input-QA, input-PROD) with the same settings;
2 Inputs work everything ok.
However, 1 of the Inputs has a very large delay in indexing the logs; And it’s always the same Input has this delay (Input-QA).
The time now is 11AM o’clock, but my logs are coming at 4AM
There was nothing wrong with it, because I created a new topic in kafka and sent the inputQA logs to it.
Yet he continued with this slowness.
Anyone have any idea what’s going on?