I have setup losgstash and graylog on a machine and filebeat on another 16 machines to send logs to graylog server via logstash.
I can see logs on graylog server for 6 of the machines but not for rest of them as log size is large(1gb each for 10 machines). Filebeat is still processing logs for these machines after running for continuously 3 days.
I have following queries regarding this scenario:
Can i interrupt the process of sending the logs from filebeat to logstash for a while so that logstash send the logs accumulated till now to graylog?
What should be the ideal speed of filebeat logs sending to logstash assuming machine which has filebeat has 16gb ram and 5gb free space and graylog server has 32gb ram and 8gb free space?
How can i be sure that logstash cache does not fill out in case of 16 machines sending logs to it continuously?
Kindly reply asap as i am working on a real project implementation.
Thanks and Regards
Sourabh Bhardwaj
+91 9899007590
little hint: You don’t need any logstash inbetween Graylog and filebeat. Graylog is able to ingest the messages itself using the Beats Input This would be one moving part less in your system
Your questions:
Yes, you can. the idea behind filebeat is that it reliably sends lines of logs and remembers where it left off the last time it stopped. Simply stop the filebeast deamon on your source machines. Filebeat has a registry of files and lines that were send.
This question I cannot really answer. It depends on your CPU, disk I/O, Network connection. But this should not be an issue to you since the beats protocol uses a concept called backpressure. If a destination is not able to process messages fast enough it will tell that to the source, which will then slow down its own processing to take of pressure of the destination system
Again, the backpressure function will take care of this
little hint: You don’t need any logstash inbetween Graylog and filebeat. Graylog is able to ingest the messages itself using the Beats Input This would be one moving part less in your system
I have used logstash in between to implement grok filter… does the same achievable without logstash?
Yes, you can. the idea behind filebeat is that it reliably sends lines of logs and remembers where it left off the last time it stopped. Simply stop the filebeast deamon on your source machines. Filebeat has a registry of files and lines that were send.
Well the real question i want to ask here is:
If i am sending large amount of data through filebeat to graylog(via logstash), it’s about 3 days but still log of a particular machine are not visible in graylog server. as per my understanding the logs are accumulated in logstash until filebeat does not find EOF or file inactive. If i want to see the logs sent till now to logstash, how can i see them?
By restarting the filebeat, stopping the filebeat, restarting the logstash or which way? I just want to see the logs accumulated in logstash from filebeat in these 3 days… how can i push logstash to send these logs to graylog so that they are visible on graylog server?
Kindly reply asap.
If i am sending large amount of data through filebeat to graylog(via logstash), it’s about 3 days but still log of a particular machine are not visible in graylog server. as per my understanding the logs are accumulated in logstash until filebeat does not find EOF or file inactive. If i want to see the logs sent till now to logstash, how can i see them?
if you miss data in Graylog you should follow the way of logs from the source to Graylog. Check all components if the messages pass-by and then proceed to the next. Until you hit Graylog.
You should check the Graylog logfile and if you find messages of this source when searching for this source and in all-messages. It might be that the system is in the past or the future - so checking time on all sending systems might be a good idea too.
This would be one moving part less in your system 
This would be one moving part less in your system