Wich log forwarder for syslog listening and output compression for a low bandwich? [resolved logstash and GELF output]

[piellick]

Hi eveyrone,

need advise for chosing a log forwarder who can listening syslog and have an output who can compress and forward to graylog.

Logstah have a syslog input and GELF output but no information about compression

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-gelf.html

SSL support is not an obligation.

Thanks

[macko003]

here is some information about GELF, all information what you need.

http://docs.graylog.org/en/2.5/pages/gelf.html

I wouldn’t use UDP only for compression.

Maybe beats output?
https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html

[piellick]

Thanks @macko003.
i have choose a different solution, using logstash with GELF Output and it’s works well.

There is an exemple for the community of a logstash configuration file for listening syslog and compress it with GELF thought UDP :

input {
  udp {
    port => 514
    type => "syslog"
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
gelf {
        host => "GRAYLOG IP"
        port => 514
    }
}

just need to install the GELF output plugin for logstash by :

logstash-plugin install logstash-output-gelf

[macko003]

ok, but you wrote

@piellick

who can compress and forward to graylog.

Based on the docs, GELF over TCP doesn’t support compression.

http://docs.graylog.org/en/2.5/pages/gelf.html

The Graylog Extended Log Format (GELF) is a log format that avoids the shortcomings of classic plain syslog:

        Limited to length of 1024 bytes – Not much space for payloads like backtraces
        No data types in structured syslog. You don’t know what is a number and what is a string.
        The RFCs are strict enough but there are so many syslog dialects out there that you cannot possibly parse all of them.
        No compression

[piellick]

Yes you right,

but by default UDP is using on logstash GELF output .

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-gelf.html#plugins-outputs-gelf-protocol