Hello.
I have log storage where gzipped Nginx access logs are stored from multiple locations (they uploaded by FTP and it’s not possible to send these logs directly to Graylog)
Currently I’m researching how I can send these logs to Graylog.
I’ve tried standard way mentioned in documentation, using netcat and raw UDP input:
zcat access.gz | netcat -u graylog_IP port
And it works well for single message, but for real log whole log content is transferred/displayed as a single record(so it’s a little bit tiresome to read 500Mb record).
Could you please advise me what will be the most efficient way to handle this issue?
Can Graylog split such input automatically? (zcat | while read | netcat works, but it’s extremely slow)
Or I should check completely different approach?
you can try loggen. but you have to uncompress it first.
zcat | loggen 127.0.0.1 1061 --read-file -
have you checked what slow at yout try? does your CPU handle the uncrompress?
I suggest to use TCP, because will send all messages, but it’s slower.
As I understand netcat is called and have to create new connection for each line - this is extremely inefficient even my server is pretty powerful.
Never heard about loggen, as I understand it’s component of syslog-ng, so it means need to switch from raw UDP to syslog TCP - I will give it a try
Graylog is not a shipper - you need to find a shipper that can work with the .gz files.
I personally would go this way: https://github.com/elastic/beats/issues/637#issuecomment-402209552
As I’m not aware of any other option that is not hacky.
Thanks @jan, I will give it a try.
Frankly speaking the real issue is not clear to me, so I’m a little bit confused in what direction to move further.
It’s not a big deal to ship with zcat and netcat if they were efficient enough, I just confused why Graylog doesn’t recognize this stream as separated records.
It’s not a big deal to ship with zcat and netcat if they were efficient enough, I just confused why Graylog doesn’t recognize this stream as separated records.
But if you create a RAW Input, you can just forward everything and each new line will be (by default) a new message.
Graylog itself will not read any files, that needs to be done by a shipper.
Solved by using Raw TCP input:
zcat log.gz | ncat localhost port
By some reason only Raw UDP input didn’t work as I expected
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.