Sending messages from Logstash to Graylog, both running on Docker

Graylog Central (peer support)
1

I have Logastash, Graylog, Elasticsearch and Mongodb each one running as docker service (I use docker-compose). I use logstash jdbc input plugin to retreive the logs and gelf output to send logs to graylog. I am able to get the logs to logstash, but fail to send them to graylog. Below is my logstash conf and graylog’s udp input.

 input {
  jdbc {
    jdbc_driver_library => "/opt/mysql-connector-java-8.0.13.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://mariadb:3306/${DB_DATABASE}"
    jdbc_user => "${DB_USERNAME}"
    jdbc_password => "${DB_PASSWORD}" 
    statement => "SELECT * FROM mdl_logstore_standard_log WHERE id > :sql_last_value ORDER BY id"
    use_column_value => true
    tracking_column => "id"
    schedule => "*/5 * * * *" ## run every 5 minutes
  }
}

filter {
  mutate {
    add_field => { "short_message" => "moodle GELF message" }
  }
}

output {
  stdout {
    id => "moodle_stdout"
    codec => json
  }
  gelf {
    id => "moodle_gelf_output"
    host => "logging_system_graylog_1"
    port => 12201
  }
}

I set the host to be the graylog’s container name, since that is the address I can ping from inside the logstash’s container. But this host and bind_address part is that I’m most unsure about.

bind_address: logging_system_graylog_1
decompress_size_limit: 8388608
override_source: <empty>
port: 12201
recv_buffer_size: 262144

I do not have any error messages in logstash's container. And the only one I have in graylog's container is the following. 
WARN : org.graylog2.plugin.inputs.transports.NettyTransport - receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=moodle_gelf, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=e6019562-4a01-471e-9373-15cce0410134} should be 262144 but is 212992.

graylog image: graylog/graylog:2.5
logstash image: logstash/logstash:6.5.4

Any help is much appreciated!
2

why not use beats/lumberjack output to Graylog?

3

Since I need to use logstash to read logs from database table, I thought it would be suitable/easier to just use logstash’s gelf output to send logs. I thought that beats is used for forwarding logs directly from the source (for example from json file), but can I use beats to forward logs from logstash to graylog? If so, then would it be better to use beats instead of logstash for forwarding logs to graylog?

4

my intention was more to use beats - the protocol/framework - as transport. As this is then TCP and include crypt of transported message.

5

but unfortunately it is not possible to read log entries from database with Beats

6

you are using the tool logstash to read messages from the database and use the output lumberhack/beats in logstash to send the messages to Graylog. That was my idea.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.