Filebeat -> Logstash -> Graylog (all SSL secured)

gruselglatz · March 28, 2019, 6:28am

Hi,

I need to setup a POC where I send Logs from 100+ filebeats to a Graylog instance, but the Graylog instance and Elasticsearch is in our HQ (not China) and the 100+ Clients are in China.
So my plan was to put an Aggregator on the China side which is logstash. Logstash should collect all filebeats and send it over to Graylog in one encrypted connection.

All connections should be encrypted, so far no problem unitl i come to the logstash -> graylog connection. The GELF output is not capable of encryption, and i dont want an extra RabbitMQ layer, when there is maybe another option.

So I dont found any doc about logstash to beats input. My question is simple: Can I send logs from logstash to the beats input in an encrypted fashion? (Or any other usefull and supported input plugin)

Sidenote: I want to use Logstash, bcs i also use other Beats there which go directly to ES.

Thank you for your help

jan · March 28, 2019, 6:49am

the “beats” output in logstash is called lumberjack - because that is the origin of the beats protocol

You might want to contact Sales ( http://graylog.org/contact-sales ) as we have something short before release that is build exactly for that. We call it Forwarder

gruselglatz · March 28, 2019, 7:34am

Hi Jan,

thanks, so you say i can use the lumberjack output to utilize the beats input?

Thanks, but I have some questions then:

Will this be a enterprise feature?
Will the forwarder support outputs to graylog and Elasticsearch directly?
Will the forwarder be able to do something like beats -> forwarder -> forwarder -> graylog

Thanks for your time!

jan · March 28, 2019, 7:38am

Logstash Lumberjack Output to BEATS Input Graylog should work flawless.

The Forwarder is Graylog Enterprise, yes (that is why I connected you to sales) and will be Graylog Forwarder Output > Graylog Forwarder Input.

gruselglatz · March 28, 2019, 7:47am

Thanks I need a Forwarder that is capable of doing multiple protocolls IN and OUT, so i will try the lumberjack way for now, and will see how the Graylog Forwarder is developing, maybe its a better option in the future then.

Thank you for the help!

jan · March 28, 2019, 8:16am

The Forwarder is a plugin to a full Graylog - so in your setup you would have one Graylog with all options in China, in that a forwarder output that includes journal (if the connection to the central is not available) and in your central the forwarder input on your Central Graylog.

gruselglatz · April 1, 2019, 10:16am

Hey Jan,

i’ve tried the lumberjack output to beats input, but it doesnt get a connection.

IDK where my problem is, i get this messages on graylog:

2019-04-01T11:28:00.230+02:00 ERROR [AbstractTcpTransport] Error in Input [Beats/5c6acc71a0303c17f682ba78] (channel [id: 0xa2af5fea, L:/172.21.2.63:5044 ! R:/172.21.2.99:46936]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 49)
2019-04-01T11:28:00.231+02:00 ERROR [AbstractTcpTransport] Error in Input [Beats/5c6acc71a0303c17f682ba78] (channel [id: 0xa2af5fea, L:/172.21.2.63:5044 ! R:/172.21.2.99:46936]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 87)
2019-04-01T11:28:49.442+02:00 ERROR [AbstractTcpTransport] Error in Input [Beats/5c6acc71a0303c17f682ba78] (channel [id: 0x8e05800f, L:/172.21.2.63:5044 ! R:/172.21.2.99:46940]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 10)

this on logstash:

Client write error, trying connect {:e=>#<IOError: Broken pipe>, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:857:in `sysread'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jruby-openssl-0.10.2-java/lib/jopenssl23/openssl/buffering.rb:57:in `fill_rbuff'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jruby-openssl-0.10.2-java/lib/jopenssl23/openssl/buffering.rb:98:in `read'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:157:in `read_version_and_type'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:145:in `ack'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:134:in `write_sync'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:42:in `write'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:65:in `flush'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:in `block in buffer_flush'", "org/jruby/RubyHash.java:1419:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:in `buffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in `buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:52:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.0.6/lib/logstash/codecs/plain.rb:40:in `encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:59:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `block in multi_receive'", "org/jruby/RubyArray.java:1792:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:390:in `block in output_batch'", "org/jruby/RubyHash.java:1419:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:389:in `output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:304:in `block in start_workers'"]}

thit is the openssl output:

openssl s_client -connect graylog.domain.com:5044 -CAfile ca-pem.crt
CONNECTED(00000003)
depth=1 DC = com, DC = domain, CN = CA
verify return:1
depth=0 CN = graylog.domain.com
verify return:1
---
Certificate chain
 0 s:/CN=graylog.domain.com
   i:/DC=com/DC=domain/CN=CA
---
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
subject=/CN=graylog.domain.com
issuer=/DC=com/DC=domain/CN=CA
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1972 bytes and written 269 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 485AB5E76E1FBECBDAF7027B180CD953A24505BFE0BF1C517BDBB3947ECCEB08
    Session-ID-ctx:
    Master-Key: 8E4A92AEFF1BB36ADF3D22331273F51D1D21AA802A988C21B787EF8CF46A7CA5BE7A3AC6837DED7869D5DF0A1CA071CE
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1554110924
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

this is the input:

bind_address: 0.0.0.0
no_beats_prefix: false
number_worker_threads: 4
override_source: <empty>
port: 5044
recv_buffer_size: 1048576
tcp_keepalive: false
tls_cert_file: /home/rene/graylog.domain.com.crt
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: true
tls_key_file: /home/rene/graylog.domain.com.rsa
tls_key_password: ********

this is the logstash config:

input {
  beats {
    port => 5050
    ssl => true
    ssl_certificate => "/home/logstash/certs/agg_hag1.crt"
    ssl_key => "/home/logstash/certs/agg_hag1-des-v1.pem"
    ssl_key_passphrase => "PWD"
    ssl_verify_mode => none
    tls_min_version => 1.2
  }
}

output {
  lumberjack {
    id => "internal_POC"
    enable_metric => true
    hosts => "graylog.domain.com"
    port => 5044
    ssl_certificate => "/home/logstash/certs/ca-pem.crt"
  }
}

gruselglatz · April 1, 2019, 10:39am

@jan is this related? https://github.com/logstash-plugins/logstash-output-lumberjack/issues/27

Is the beats input only capable of lumberjack V2?

jan · April 1, 2019, 11:12am

to be honest - I do not know.

Sorry didn’t have the time to create a POC for that and check.

gruselglatz · April 2, 2019, 7:43am

Ok Thanks, but do you have a config for the lumberjack output to beats input that worked or is working?

I ask, because you told me in this thread, that it should work. So maybe I get the push into the right direction with a working config.

Thanks for your time.

system · April 16, 2019, 7:43am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat configuration Graylog Central (peer support) sidecar , filebeat-linux	46	49933	July 2, 2018
Adding Graylog to already existing ELK stack Graylog Central (peer support)	5	5574	March 26, 2018
Not able to send logs from filebeat to graylog Graylog Central (peer support)	7	3253	July 30, 2020
Metricbeat not working through Logstash Graylog Central (peer support)	5	1091	September 24, 2019
Graylog 3.1.4 - Containerized \| Elasticsearch-oss 6.8.6 - on host \| Filebeat 6.8.5 - on host Graylog Central (peer support) sidecar , nxlog , filebeat-linux , filebeat-windows , winlogbeat , nodatanx , nosendlogfblx , clfblx	8	2552	March 13, 2020

Filebeat -> Logstash -> Graylog (all SSL secured)

Related topics