Unable to send message over Raw/Plaintext TCP input

[umeshdc]

I am trying to use logstash TCP output in order to send the log messages to graylog via Raw/Plaintext TCP input. But unable to see the log messages on graylog UI even though I can observe the incoming traffic to my input.

Please help in finding the issue with “Raw/Plaintext TCP input” or any link how to use “Raw/Plaintext TCP input” will also help us in order to follow that.

Thanks in Advance
Umesh DC

[jochen]

Please provide the complete configuration of Logstash and the Graylog input(s).

[umeshdc]

@jochen thanks for quick replay.

Please find graylog configurations as below,

# WARNING: Maintained by Puppet, manual changes will be lost!

allow_highlighting = true
allow_leading_wildcard_searches = true
content_packs_dir = /usr/share/graylog-server/contentpacks
elasticsearch_hosts = http://XXXXXX.com:9200,http://XXXXXX.com:9200
elasticsearch_index_prefix = graylog2
elasticsearch_max_number_of_indices = 20
elasticsearch_max_time_per_index = 1d
elasticsearch_replicas = 1
is_master = true
message_journal_dir = /var/lib/graylog-server/journal
mongodb_uri = mongodb://admin:admin@XXXXXX.com:27017/graylog2
outputbuffer_processors = 24
password_secret = XXXXXX
plugin_dir = /usr/share/graylog-server/plugin
processbuffer_processors = 40
rest_listen_uri = http://0.0.0.0:9000/api/
root_password_sha2 = XXXXXX
root_timezone = Europe/Berlin
root_username = admin
rotation_strategy = time
web_enable = true
web_endpoint_uri = http://<graylogserver.com>/api/
web_listen_uri = http://0.0.0.0:9000/

Input Details:

Name: logstash-plain-test-tcp 
Type: Raw/Plaintext TCP 
Status: RUNNING
bind_address: 0.0.0.0
max_message_size: 2097152
override_source: <empty>
port: 12221
recv_buffer_size: 1048576
tcp_keepalive: false
tls_cert_file: <empty>
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: false
tls_key_file: <empty>
tls_key_password: ********
use_null_delimiter: false 

Logstash config file that is used to forward the log is as below,

input {
    file {
            type => app
            path => ["/var/log/test.log"]
        }
}
output {
    tcp {
        host => "<graylog_hostname>"
        port => <graylog input port number>
    }
}

Log messages are reaching graylog input and can be seen in Throughput / Metrics corresponding to input on graylog console. But messages are not populating on graylog dashboard.

Need your help to debug the issue.

Thanks

[jochen]

@umeshdc

output {
    tcp {
        host => "<graylog_hostname>"
        port => <graylog input port number>
    }
}

What are the specific values in your Logstash configuration?

[umeshdc]

@jochen Currently i am using the graylog IP address itself as host as below,

output {
   tcp {
       host => "My Graylog IP address"
       port => 12221
   }
}

[jochen]

What happens when you run the following command on the machine running Logstash?

# echo "Test message" | nc <Graylog IP address> 12221

Are there any error messages in the logs of your Graylog node?

[umeshdc]

@jochen This command works, I am able to see the logged message on Graylog Dashboard.

[jochen]

So you know the problem isn’t Graylog but probably Logstash.

You can use the Logstash stdout output to further investigate your Logstash configuration:
https://www.elastic.co/guide/en/logstash/6.x/plugins-outputs-stdout.html

Be aware that Logstash will only send new lines appended to /var/log/test.log by default (see https://www.elastic.co/guide/en/logstash/6.x/plugins-inputs-file.html#plugins-inputs-file-start_position for details).

[umeshdc]

@jochen Thanks for continuous support in order to solve my problem. My issue is fixed.

Issue was from logstash side, where Logstash TCP output was sending log messages in lines so it graylog was unable to handle the messages. So added codec => json_lines in logstash config file and it is working as expected,

output {
    tcp {
        **codec => json_lines**
        host => “<Graylog Hostname>"
        port => 12221
    }
}

Thanks for the support and we can close this ticket.