Sending messages from Logstash to Graylog, both running on Docker

I have Logastash, Graylog, Elasticsearch and Mongodb each one running as docker service (I use docker-compose). I use logstash jdbc input plugin to retreive the logs and gelf output to send logs to graylog. I am able to get the logs to logstash, but fail to send them to graylog. Below is my logstash conf and graylog’s udp input.

 input {
  jdbc {
    jdbc_driver_library => "/opt/mysql-connector-java-8.0.13.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://mariadb:3306/${DB_DATABASE}"
    jdbc_user => "${DB_USERNAME}"
    jdbc_password => "${DB_PASSWORD}" 
    statement => "SELECT * FROM mdl_logstore_standard_log WHERE id > :sql_last_value ORDER BY id"
    use_column_value => true
    tracking_column => "id"
    schedule => "*/5 * * * *" ## run every 5 minutes
  }
}

filter {
  mutate {
    add_field => { "short_message" => "moodle GELF message" }
  }
}

output {
  stdout {
    id => "moodle_stdout"
    codec => json
  }
  gelf {
    id => "moodle_gelf_output"
    host => "logging_system_graylog_1"
    port => 12201
  }
}

I set the host to be the graylog’s container name, since that is the address I can ping from inside the logstash’s container. But this host and bind_address part is that I’m most unsure about.

bind_address: logging_system_graylog_1
decompress_size_limit: 8388608
override_source: <empty>
port: 12201
recv_buffer_size: 262144

I do not have any error messages in logstash's container. And the only one I have in graylog's container is the following.
WARN : org.graylog2.plugin.inputs.transports.NettyTransport - receiveBufferSize (SO_RCVBUF) for input GELFUDPInput{title=moodle_gelf, type=org.graylog2.inputs.gelf.udp.GELFUDPInput, nodeId=e6019562-4a01-471e-9373-15cce0410134} should be 262144 but is 212992.

graylog image: graylog/graylog:2.5
logstash image: logstash/logstash:6.5.4

Any help is much appreciated!

why not use beats/lumberjack output to Graylog?

Since I need to use logstash to read logs from database table, I thought it would be suitable/easier to just use logstash’s gelf output to send logs. I thought that beats is used for forwarding logs directly from the source (for example from json file), but can I use beats to forward logs from logstash to graylog? If so, then would it be better to use beats instead of logstash for forwarding logs to graylog?

my intention was more to use beats - the protocol/framework - as transport. As this is then TCP and include crypt of transported message.

but unfortunately it is not possible to read log entries from database with Beats

you are using the tool logstash to read messages from the database and use the output lumberhack/beats in logstash to send the messages to Graylog. That was my idea.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.