Unknown beats protocol version: 22, 3, 71, 69

USaman · March 25, 2022, 8:48pm

Hello, I am new to Graylog. I have implemented Beats (filebeat heartbeat and metricbeat) on Remote Nomad cluster. I am shipping logs and metric to a centralised server. Everything seems to work fine until I encountered with this error.

Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0x6ae4d106, L:/172.27.0.4:5044 ! R:/10.142.3.97:33394]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 22)
graylog_1        | 2022-03-25 20:01:08,737 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0x6ae4d106, L:/172.27.0.4:5044 ! R:/10.142.3.97:33394]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 3)
graylog_1        | 2022-03-25 20:02:00,181 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0xb16d8fbf, L:/172.27.0.4:5044 ! R:/10.142.3.97:33492]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 22)
graylog_1        | 2022-03-25 20:02:00,182 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0xb16d8fbf, L:/172.27.0.4:5044 ! R:/10.142.3.97:33492]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 3)
graylog_1        | 2022-03-25 20:02:51,222 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0xff6c3305, L:/172.27.0.4:5044 ! R:/10.142.3.97:33588]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 22)
graylog_1        | 2022-03-25 20:02:51,222 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0xff6c3305, L:/172.27.0.4:5044 ! R:/10.142.3.97:33588]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 3)
graylog_1        | 2022-03-25 20:03:50,868 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0x1feedc93, L:/172.27.0.4:5044 ! R:/10.142.3.97:33700]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 22)
graylog_1        | 2022-03-25 20:03:50,868 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0x1feedc93, L:/172.27.0.4:5044 ! R:/10.142.3.97:33700]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 3)
graylog_1        | 2022-03-25 20:04:21,675 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0xc777ace6, L:/172.27.0.4:5044 ! R:/10.142.3.97:33768]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 22)
graylog_1        | 2022-03-25 20:04:21,675 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0xc777ace6, L:/172.27.0.4:5044 ! R:/10.142.3.97:33768]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 3)
graylog_1        | 2022-03-25 20:04:59,062 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0xcc0387cd, L:/172.27.0.4:5044 ! R:/10.142.3.97:33836]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 22)
graylog_1        | 2022-03-25 20:04:59,062 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0xcc0387cd, L:/172.27.0.4:5044 ! R:/10.142.3.97:33836]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 3)
graylog_1        | 2022-03-25 20:05:37,214 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0x27d1622d, L:/172.27.0.4:5044 ! R:/10.142.3.97:33914]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 22)

Currently I am using
Beats: 7.10.2
Elasticsearch: 7.10.2
Graylog: 4.2.7

I am not using Logstash. Logs are directly shipped to Graylog server. TLS is disabled on both side. I am stuck here since weeks now. Can someone please help me.

tmacgbay · March 25, 2022, 9:39pm

I saw something as I was searching that there could be an issue in an extractor attached to the beats input - You can post the extractor configurations if you don’t find the problem there… @gsmith is better with extractors than I am… just sayin…

USaman · March 25, 2022, 9:53pm

Hello, Thank you for quick response. Correct me if I am wrong, extractor comes after there are input messages on the Graylog server. I don’t have an extractor configured as I am figuring out how to get messages first.

As you can see from the screenshot there are no messages just active connection and above I posted the error I am getting “unkown beat protocol version”

gsmith · March 25, 2022, 11:15pm

Hello @USaman

I might be able to help. what I understand from this error

(cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 3)

something from that client is still sending invalid Beats packets to Graylog’s Beats input. You might want to investigate further.

Troubleshooting:

When you click on show received messages. button, Look at the timestamp also enable your refresh rate to like 2 seconds or something similar. Sometime the Date/time are off you may not see message for another 6-8 hours, just a thought.

I’m unsure how FileBeat was configured. Show that may help

How long was everything working fine ( day, months, hour)?

The extractor modify message on that input so when you click show received message button the fields are present. Elasticsearch will take care of that. It is possible that an extractor/s is bad or Not working properly, it could drop or stop messages. Hence " No messages shown"

Perhaps try just FileBeat and see if you get the same errors, if so do the same with MetricBeat. If bothof them we would need to see how you configured your environment, if possible.

By chance are you using Docker?

If so this might be where the issue is. Then we would need to see you configuration of this. Make sure you port are not blocked,

EDIT:
Is this you also?

I also found this post, It may help even thou its a different version then yours, Same problem.

"Unknown beats protocol version: 3" using winlogbeat v5.1.1

EDIT2:

I Just noticed something. You stated

In your screenshot that Beat Input configuration is incorrect. Try the following

USaman · March 25, 2022, 11:59pm

Hello, Yes everything is running on Docker. I checked “show messages” Clicked on “update every 2 seconds” I see nothing.

I am sorry I think I wrote it in a wrong way. What I meant by everything was working fine is when I saw filebeat is able to connect to the graylog server (that when I thought everything is working fine until I saw graylog server logs). I never got any messages ever. Just the active connection.

Here is my filebeat.hcl file running on Nomad

job "filebeat" {
  datacenters = ["DC"]

  type = "system"
  update {
    min_healthy_time = "10s"
    healthy_deadline = "5m"
    progress_deadline = "10m"
    auto_revert = true
  }
  
  group "filebeat" {
    task "filebeat" {
      driver = "docker"

      config {
        image = "docker.elastic.co/beats/filebeat:7.10.2"
        args = [
          "-c", "/local/filebeat.yml",
          "--path.data", "/alloc/data/filebeat",
          "--path.logs", "/alloc/logs",
        ]
        mount {
          type     = "bind"
          source   = "local/filebeat.yml"
          target   = "/usr/share/filebeat/filebeat.yml"
          readonly = true  
        }
      } 
      template {
        data = <<template
  filebeat.inputs:
    - paths:
        - /alloc/logs/*.stdout.[0-9]*
      exclude_files: ['\.fifo$']  
      type: filestream
      scan_frequency: 1s
      fields_under_root: true
      fields:
        app: ${NOMAD_JOB_NAME}
    
    - paths:
        - /alloc/logs/*.stderr.[0-9]*
      exclude_files: ['\.fifo$']  
      type: filestream
      scan_frequency: 15s
      fields_under_root: true
      fields:
        app: ${NOMAD_JOB_NAME}
    
    - paths:
        - /alloc/logs/filebeat
      exclude_files: ['\.fifo$']  
      type: filestream
      scan_frequency: 1s
      fields_under_root: true
      fields:
        app: ${NOMAD_JOB_NAME}
  
      encoding: utf-8    
      backoff: 1s
      close_eof: false
      close_inactive: 5m
      close_removed: true

      filebeat.registry.path: ${path.data}/registry
      filebeat.registry.file_permissions: 0600
      filebeat.registry.flush: 0s

  output.logstash:
    hosts: ["graylog-server:5044"]
    index: "[filebeat-]8.1.1-YYYY.MM.DD"
    protocol: http
    tls: disable
    ssl.verification_mode : none
    bulk_max_size: 1024



  logging.level: debug
    
      template
      destination = "local/filebeat.yml"
      } 
    }
  }
}

This is my docker-compose.yml

###################################
  # Greylog container logging start #
  ###################################
  # Taken from https://docs.graylog.org/en/4.0/pages/installation/docker.html
  # MongoDB: https://hub.docker.com/_/mongo/
version: '3.7'
services:
  mongo:
    image: mongo:3
    user: root
    networks:
      - graylog
    ports:
      - 27017:27017
  elasticsearch:
    user: root
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.2
    ports:
      - 9200:9200
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - type: bind
        source: /srv/docker/elasticsearch/elasticsearch.yml
        target: /usr/share/elasticsearch/config/elasticsearch.yml

      - type: bind
        source: /srv/docker/elasticsearch/limits.conf
        target: /etc/security/limits.conf

      - esdata:/usr/share/elasticsearch/data:rw


    networks:
      - graylog
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
 graylog:
    image: graylog/graylog:4.2.7
    user: root
    volumes:
      - type: bind
        source: /srv/docker/graylog/graylog.conf
        target: /usr/share/graylog/data/config/graylog.conf
    environment:
      - GRAYLOG_ELASTICSEARCH_VERSION=7
      - GRAYLOG_REST_LISTEN_URI=https://0.0.0.0:9000/api/
    networks:
      - graylog
    links:
      - mongo:mongo
      - elasticsearch
    depends_on:
      - mongo
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      #Beats
      - 5044:5044
  ###################################
  # Greylog container logging end   #
  ###################################
networks:
  graylog:
    driver: bridge
volumes:
  esdata:
    driver: local

For now only filebeat is sending logs to graylog but I tried sending metrics via metricbeat and heartbeat I got same error sometimes “unknown beats protocol version:” 22 or 71 or 69 even 3.

PS: Yes I posted the same issue on elasticsearch.

gsmith · March 26, 2022, 12:01am

Give me sec , I try to write up something for you.

gsmith · March 26, 2022, 12:02am

Oh I see, How about adjusting your Input as shown above?

USaman · March 26, 2022, 12:18am

Tried the above beats configuraton getting same error

raylog_1        | 2022-03-26 00:15:21,487 INFO : org.graylog2.inputs.InputStateListener - Input [Beats/623e1f6f06d36604070d9aee] is now STARTING
graylog_1        | 2022-03-26 00:15:21,489 WARN : org.graylog2.plugin.inputs.transports.AbstractTcpTransport - receiveBufferSize (SO_RCVBUF) for input Beats2Input{title=Beats, type=org.graylog.plugins.beats.Beats2Input, nodeId=319dece7-6707-44d0-90a0-67d1d5092d89} (channel [id: 0x0f686c74, L:/0.0.0.0:5044]) should be >= 1048576 but is 425984.
graylog_1        | 2022-03-26 00:15:21,492 INFO : org.graylog2.inputs.InputStateListener - Input [Beats/623e1f6f06d36604070d9aee] is now RUNNING
graylog_1        | 2022-03-26 00:15:41,192 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0x2914a4bc, L:/172.27.0.4:5044 ! R:/10.142.3.97:45712]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 22)
graylog_1        | 2022-03-26 00:15:41,193 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0x2914a4bc, L:/172.27.0.4:5044 ! R:/10.142.3.97:45712]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 3)
graylog_1        | 2022-03-26 00:16:24,264 ERROR: org.graylog2.plugin.inputs.transports.AbstractTcpTransport - Error in Input [Beats/623e1f6f06d36604070d9aee] (channel [id: 0x2dd442c5, L:/172.27.0.4:5044 ! R:/10.142.3.97:45794]) (cause io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: Unknown beats protocol version: 22)

gsmith · March 26, 2022, 12:36am

Ok. So, with the added info you gave us this is what I could find for you.

First, you have FileBeat on your local machine.
Second, you’re trying to receive logs from your remote devices but you input is ONLY pointing to you localhost.

FYI, then you will need to use a Global setting in your INPUT not your node. Shown from the screenshot above

I just started using Docker past couple weeks, but I noticed your using an old configuration AND you using HTTPS without certificates. That would be a bad idea.

GRAYLOG_REST_LISTEN_URI=https://0.0.0.0:9000/api/

Perhaps something like this, this is mine,

version: '2'
services:
   # MongoDB: https://hub.docker.com/_/mongo/
  mongodb:
    image: mongo:4
    network_mode: bridge
   # DB in share for persistence
    volumes:
      - mongo_data:/data/db
   # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/6.6/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
    network_mode: bridge
    #data folder in share for persistence
    volumes:
      - es_data:/usr/share/elasticsearch/data
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
   # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:4.2-jre11
    network_mode: bridge
    dns:
      - 192.168.2.15
      - 192.168.2.16
   # journal and config directories in local NFS share for persistence
    volumes:
      - graylog_journal:/usr/share/graylog/data/journal
      - graylog_bin:/usr/share/graylog/bin
      - graylog_data:/usr/share/graylog/data
    environment:
      # Container time Zone
      - TZ=America/Chicago
      # CHANGE ME (must be at least 16 characters)!
      - GRAYLOG_PASSWORD_SECRET=pJod1TRZAckHmqM2oQPqX1qnLVJS99jHm2DuCux2Bpiuu2XLTZuyb2YW9eHiKLTifjy7cLpeWIjWgMtnwZf6Q79HW2nonDhN
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=ef92b778bafe771e89245b89ecbc08a44a4e166c06659911881f383d4473e94f
      - GRAYLOG_HTTP_BIND_ADDRESS=0.0.0.0:9000
      - GRAYLOG_HTTP_EXTERNAL_URI=http://192.168.1.28:9000/
      - GRAYLOG_ROOT_TIMEZONE=America/Chicago
      - GRAYLOG_ROOT_EMAIL=greg.smith@domain.com
      - GRAYLOG_HTTP_PUBLISH_URI=http://192.168.1.28:9000/
      - GRAYLOG_TRANSPORT_EMAIL_PROTOCOL=smtp
      - GRAYLOG_HTTP_ENABLE_CORS=true
      - GRAYLOG_TRANSPORT_EMAIL_WEB_INTERFACE_URL=http://192.168.1.28:9000/
      - GRAYLOG_TRANSPORT_EMAIL_HOSTNAME=192.168.1.28
      - GRAYLOG_TRANSPORT_EMAIL_ENABLED=true
      - GRAYLOG_TRANSPORT_EMAIL_PORT=25
      - GRAYLOG_TRANSPORT_EMAIL_USE_AUTH=false
      - GRAYLOG_TRANSPORT_EMAIL_USE_TLS=false
      - GRAYLOG_TRANSPORT_EMAIL_USE_SSL=false
      - GRAYLOG_TRANSPORT_FROM_EMAIL=root@localhost
      - GRAYLOG_TRANSPORT_SUBJECT_PREFIX=[graylog]
      - GRAYLOG_REPORT_DISABLE_SANDBOX=true      
    links:
      - mongodb:mongo
      - elasticsearch
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 8514:8514
      # Syslog UDP
      - 8514:8514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp
      # Reports
      - 9515:9515
      - 9515:9515/udp
      # email
      - 25:25
      - 25:25/udp
      # web
      - 80:80
      - 443:443
      # beats
      - 5044:5044
      - 5044:5044/udp
      - 5044:5055
      - 5055:5055/udp
#Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
volumes:
  mongo_data:
    driver: local
  es_data:
    driver: local
  graylog_journal:
    driver: local
  graylog_bin:
    driver: local
  graylog_data:
    driver: local

As you can see the GRAYLOG_REST_LISTEN_URI=https://0.0.0.0:9000/api/ is no longer valid. And if it work , I’m wondering how.

We both have similar settings my lab Docker has more since I’m learning and also have the enterprise version enabled.

Please take a look at this documentation , that is if you haven’t already.

Docker - Installing Graylog

Sum it up
Your Docker yaml file I believe has incorrect and/or old configurations. If its working now it might not later, just a thought.

I would use a Global configuration on your inputs for now since your just starting out.

Making sure the container/s is healthy.

docker ps

Another idea is to use FileBeat on localhost and the other beats on a different input, just change the port to something like 5055, just an idea.

gsmith · March 26, 2022, 12:43am

Your docker-compose.yml

Try adding

 # beats
      - 5044:5044/udp <--- this
      - 5055:5055/udp

docker-compose up -d --build

USaman · March 26, 2022, 1:03am

Thank you for such a detailed insight. You actually pointed out so many mistakes which I would have never noticed in million years! I have noted all your points. I will implement it and update. Thank you for your time.

gsmith · March 26, 2022, 1:09am

No problem, What I notice was your logs you showed above, Right now practicing my Docker and I was like those look like my container files , I wonder if they are using Docker, Bingo you were. Docker and Hyper-v virtual machines have the same characteristics. Its a little difficult but I’m getting the hang of it.

USaman · March 26, 2022, 11:30pm

Solved the issue.

output.logstash:
      hosts: ["lP:5046"]

Output section of my logstash in Beats had so many things like TLS dsiable and index etc etc that should NOT be there. I removed everything extra and everything worked fine until I ran out of JVM memory

system · April 9, 2022, 11:31pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unknown Beats protocol Graylog Central (peer support) pipeline-rules , debuggingpl	7	2665	June 13, 2018
SSL issues with beats input Graylog Central (peer support) sidecar , winlogbeat	3	1691	September 30, 2020
Graylog INPUT Beat Graylog Central (peer support) sidecar , nxlog , filebeat-linux , filebeat-windows , winlogbeat , nodatanx , nosendlogfblx , clfblx	10	5470	July 31, 2020
ERROR [NettyTransport] Error in Input [Beats/] Graylog Central (peer support)	2	439	March 4, 2019
Packebeat Input Graylog Central (peer support) sidecar , nxlog , filebeat-windows , winlogbeat , nodatanx	3	1715	September 11, 2019

Unknown beats protocol version: 22, 3, 71, 69

Related Topics