Graylog rsync issue

Hi All,
I am using a centralised server (CS) where I rsync the logs from all applications every 5 minutes. I use filebeat on the centralised server to send logs to elastic search.
Issue I face is that the log on my hosts rotate after 1GB and hence it almost gets rotated every hour. When I rsync the same I face issue that on the CS that there is no log rotation done here.

For eg: p1dm.log is the filename which gets rotated to p1dm-data-timestamp.log after 1 hour and then new p1dm.log is created.

On my CS rsync will collect logs and it already has a p1dm.log file which is continuously being written to. Once the new p1dm.log gets created on my hosts, will it still continue to write in my same p1dm.log file on CS server. If yes then that file will get huge . Any suggestions on what can I do so that my rsync syncs files along with logrotation.

Thanks in advance for help.

he @SuhasMUFC

your question is not really a Graylog question - but we will try to help with that.

What does your rsync command look like? Did you run all this by some bash scripts? How do they look?

The other question - why did you have decided for this kind of transport model? You could have used filebeat on each sender and if a proxy is needed use logstash or nginx to transport the messages from one network to the other.

Hi Jan,

Issue is I do not want filebeat on my production servers as it could take up CPU and my server could hang in rare situations. We have a rule in our company wherein we do not run anything like filebeat etc on prod servers. rsync is being done via a shell script. Logging is very high with a single server producing 70gb logs per day and i have multiple servers like this .

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.