Graylog 3.0.1 - Unknown beats protocol version

Arthur · June 3, 2019, 7:11pm

Hi,

Thanks in advance for your help. I had been using smoothly Graylog 3.0.1 during the last May without any problem. However, On May 31st started receiving a new error Unknown beats protocol version. I have not conducted any change during May which makes more difficult to figure out the source of the problem. Searching I found that this might be related to certificates.

The following is complete error I am getting:

2019-06-03 11:34:34,078 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/5c92d281389808436d8b6198] (channel [id: 0x286f1cac, L:/XX.XX.XX.XX:5044 ! R:/XX.XX.XX.XX:40724]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 22)
2019-06-03 11:34:34,080 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/5c92d281389808436d8b6198] (channel [id: 0x286f1cac, L:/XX.XX.XX.XX:5044 ! R:/XX.XX.XX.XX:40724]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 3)

I have the following configuration regarding certificates:

#Generate a self-signed private key/certificate
openssl req -newkey rsa:2048 -x509 -sha256 -nodes -days 3650 -config /app/graylog/3.0.1/config/openssl-graylog.cnf -keyout /app/graylog/3.0.1/config/pkcs5-plain.pem  -out /app/graylog/3.0.1/config/graylog-cert.pem -batch

#Convert PKCS#5 key to PKCS#8
openssl pkcs8 -in /app/graylog/3.0.1/config/pkcs5-plain.pem -topk8 -out /app/graylog/3.0.1/config/graylog-key.pem -passout pass:secret

server.conf:

################
# HTTPS settings
################

#### Enable HTTPS support for the HTTP interface
#
# This secures the communication with the HTTP interface with TLS to prevent request forgery and eavesdropping.
#
# Default: false
http_enable_tls = True

# The X.509 certificate chain file in PEM format to use for securing the HTTP interface.
http_tls_cert_file = /app/graylog/3.0.1/config/graylog-cert.pem

# The PKCS#8 private key file in PEM format to use for securing the HTTP interface.
http_tls_key_file = /app/graylog/3.0.1/config/graylog-key.pem

# The password to unlock the private key used for securing the HTTP interface.
http_tls_key_password = secret

# List of Elasticsearch hosts Graylog should connect to.
# Need to be specified as a comma-separated list of valid URIs for the http ports of your elasticsearch nodes.
# If one or more of your elasticsearch hosts require authentication, include the credentials in each node URI that
# requires authentication.
#
# Default: http://127.0.0.1:9200
elasticsearch_hosts = http://127.0.0.1:9200

# settings to be passed to elasticsearch's client (overriding those in the provided elasticsearch_config_file)
# all these
# this must be the same as for your Elasticsearch cluster
elasticsearch_cluster_name = graylog

Beats configuration:
BeatsInput

Currently, the filebeat is shipping successfully the messages, but Graylog is not able to get them. In the graylog logs just Error in Input error is shown.

This are the versions of the components:

Graylog 3.0.1
Mongo 4.0.6
Elasticsearch 6.6.0
Filebeat 6.6.0

Do you have any idea what be wrong?

Thanks in advance for your help.

Cheers,
Arturo

jan · June 4, 2019, 6:56am

what is the last you have changed before the error happens?

Think of any package update or cable movement you have done in your infrastructure.

Arthur · June 4, 2019, 9:27pm

Hi Jan,

Thanks for your reply, I do not have any packages updates, I have checked with the following commands, but not finding any package recently installed/deleted:

yum history list
yum history list kernel

This environment is in Azure, I have the required ports listening:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      128019/mongod
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      6284/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      6414/master
tcp6       0      0 10.99.2.19:9000         :::*                    LISTEN      38713/java
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd
tcp6       0      0 :::9200                 :::*                    LISTEN      128155/java
tcp6       0      0 :::9300                 :::*                    LISTEN      128155/java
tcp6       0      0 :::5044                 :::*                    LISTEN      38713/java
tcp6       0      0 :::22                   :::*                    LISTEN      6284/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      6414/master

Filebeat server can ping the other server hosting Graylog, mongo and ES.

I see Filebeat is able to collect the logs:

{"source":"/apps/logs/APP-api/APP-api.log","offset":72508669,"timestamp":"2019-06-04T09:58:45.607831179-07:00","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":54656436,"device":2050}}

I am aware of Beats plugin as the main entrance to get files from Filebeat. Then, I assume the messages are not getting/processed by Beats plugin. Can I find logs specifically for Beats? I am just getting the logs I provided in the first comment.

Thanks,
Arturo

mpfz0r · June 5, 2019, 7:53am

Hi,

this looks like your filebeat is running with TLS enabled,
while your Input is configured without.
Unfortunately there is no auto-detection for TLS on our Input.

Arthur · June 5, 2019, 5:08pm

Thanks for your reply Marco, I will take a look at that.

I found out that the errors Unknown beats protocol version are related to the extractors configured in Beats. I have been adding more logs to analyze in Filebeat and it seems that the existing rules in the extractors were not longer valid. I updated the grok patterns and I am getting the logs right away and the errors do not appear in Graylog logs.

Thanks for your help

mpfz0r · June 7, 2019, 8:50am

That’s good to hear @Arthur

Do you mind sharing more details about what actually caused this?
Can you try to reproduce this error again?
From my understanding the Unknown beats protocol version error would be triggerd in the Input,
which is before any extractors come into play.

Thanks,
Marco

system · June 21, 2019, 9:01am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unknown beats protocol version: 22, 3, 71, 69 Graylog Central (peer support) elastic	13	3636	April 9, 2022
Unknown Beats protocol Graylog Central (peer support) pipeline-rules , debuggingpl	7	2753	June 13, 2018
Graylog TLS client authentication Unknown beats protocol version Graylog Central (peer support)	2	525	December 27, 2022
Unknown beats protocol version Graylog Central (peer support)	2	698	April 29, 2021
Looking for help with some errors we're receiving on our Windows TLS - [AbstractTcpTransport] Graylog Central (peer support)	4	520	October 18, 2023

Graylog 3.0.1 - Unknown beats protocol version

Related topics