Metricbeats Communication Issue to Graylog

Graylog Central (peer support)
1

Hi All,
I am attempting to set up Metricbeat on my Graylog instance and am having initial problems getting it to connect. I have installed metricbeat on a windows machine and linux machine and neither can connect to graylog. I do have a Beats input created within graylog, running on the configured port (5043). Please see below for component versions and output from linux machine that is having issues. Any insights are greatly appreciated.

graylog-server.noarch 2.4.3-1 @graylog
elasticsearch.noarch 2.4.6-1 @elasticsearch-2.x
mongodb-org.x86_64 3.6.3-1.el7 @mongodb-org-3.6

tail -f /var/log/metricbeat/metricbeat

2018-05-11T13:32:47.456-0700 INFO elasticsearch/client.go:145 Elasticsearch url: http://162.70.27.206:5043
2018-05-11T13:32:47.458-0700 INFO pipeline/module.go:76 Beat name: lin-play-02
2018-05-11T13:32:47.459-0700 INFO instance/beat.go:301 metricbeat start running.
2018-05-11T13:32:47.461-0700 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-05-11T13:32:47.462-0700 INFO cfgfile/reload.go:127 Config reloader started
2018-05-11T13:32:47.463-0700 INFO cfgfile/reload.go:258 Starting 3 runners …
2018-05-11T13:32:47.463-0700 INFO cfgfile/reload.go:219 Loading of config files completed.
2018-05-11T13:32:49.477-0700 ERROR pipeline/output.go:74 Failed to connect: Get http://162.70.27.206:5043: dial tcp 162.70.27.206:5043: getsockopt: connection refused
2018-05-11T13:32:51.478-0700 ERROR pipeline/output.go:74 Failed to connect: Get http://162.70.27.206:5043: dial tcp 162.70.27.206:5043: getsockopt: connection refused
2018-05-11T13:32:55.478-0700 ERROR pipeline/output.go:74 Failed to connect: Get http://162.70.27.206:5043: dial tcp 162.70.27.206:5043: getsockopt: connection refused
2018-05-11T13:33:03.480-0700 ERROR pipeline/output.go:74 Failed to connect: Get http://162.70.27.206:5043: dial tcp 162.70.27.206:5043: getsockopt: connection refused
^C

service metricbeat status

● metricbeat.service - metricbeat
Loaded: loaded (/usr/lib/systemd/system/metricbeat.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2018-05-11 13:32:47 MST; 21s ago
Docs: https://www.elastic.co/guide/en/beats/metricbeat/current/index.html
Main PID: 23655 (metricbeat)
CGroup: /system.slice/metricbeat.service
└─23655 /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -path.home /usr/share/metricbeat -path.config /etc/metricbeat -path.data /v…

May 11 13:32:47 lin-play-02 systemd[1]: Started metricbeat.
May 11 13:32:47 lin-play-02 systemd[1]: Starting metricbeat…

2

What’s the complete configuration of Metricbeat?
What’s the complete configuration of the Beats input in Graylog?
What’s the output of the following commands on the machine running Graylog?

# sudo netstat -tupeln
# sudo iptables -Ln
3

Hi Jochen, thanks for the reply. Here is the requested info.
Metricbeat.yml:

###################### Metricbeat Configuration Example #######################

# This file is an example configuration file highlighting only the most common
# options. The metricbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html

#==========================  Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging


#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

#============================= Elastic Cloud ==================================

# These settings simplify using metricbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["162.70.27.206:5043"]
  index: 'my-index-%{+yyyy.MM.dd}'
setup.template.name: "my-index"
setup.template.pattern: "my-index-*"

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

#============================== Xpack Monitoring ===============================
# metricbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#xpack.monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well. Any setting that is not set is
# automatically inherited from the Elasticsearch output configuration, so if you
# have the Elasticsearch output configured, you can simply uncomment the
# following line.
#xpack.monitoring.elasticsearch:

Metricbeat Input in Graylog:
Editing Input Metric Beats 5043

Global
Should this input start on all nodes
Node
On which node should this input start
Title
Metric Beats 5043

Bind address
0.0.0.0
Address to listen on. For example 0.0.0.0 or 127.0.0.1.

Port
5043
Port to listen on.

Receive Buffer Size(optional)
1048576
The size in bytes of the recvBufferSize for network connections to this input.

TLS cert file(optional)
Path to the TLS certificate file

TLS private key file(optional)
Path to the TLS private key file

Override source(optional)
advadmin
The source is a hostname derived from the received packet by default. Set this if you want to override it with a custom string.

TLS key password(optional)
••••••••••••••
The password for the encrypted key file.

TLS client authentication(optional)
Whether clients need to authenticate themselves in a TLS connection

TLS Client Auth Trusted Certs(optional)
TLS Client Auth Trusted Certs (File or Directory)

TCP keepalive(optional)
Enable TCP keepalive packets

Enable TLS(optional)
Accept TLS connections

Netstat & Iptables Output:

graylog>>/root>>sudo netstat -tupeln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      988        53260      8552/mongod
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      0          22037      1/systemd
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      0          10818      1988/dnsmasq
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          10683      1582/sshd
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      0          33002      1578/cupsd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      0          33024      1969/master
tcp        0      0 0.0.0.0:514             0.0.0.0:*               LISTEN      0          17606      1103/rsyslogd
tcp6       0      0 162.70.27.206:9000      :::*                    LISTEN      984        14312      8392/java
tcp6       0      0 :::12201                :::*                    LISTEN      984        51051      8392/java
tcp6       0      0 :::111                  :::*                    LISTEN      0          22036      1/systemd
tcp6       0      0 127.0.0.1:9200          :::*                    LISTEN      985        37659      8420/java
tcp6       0      0 ::1:9200                :::*                    LISTEN      985        37658      8420/java
tcp6       0      0 :::80                   :::*                    LISTEN      0          18659      1609/httpd
tcp6       0      0 :::5043                 :::*                    LISTEN      984        306002     8392/java
tcp6       0      0 :::5044                 :::*                    LISTEN      984        28347      8392/java
tcp6       0      0 127.0.0.1:9300          :::*                    LISTEN      985        14273      8420/java
tcp6       0      0 ::1:9300                :::*                    LISTEN      985        14270      8420/java
tcp6       0      0 :::22                   :::*                    LISTEN      0          10685      1582/sshd
tcp6       0      0 ::1:631                 :::*                    LISTEN      0          33001      1578/cupsd
tcp6       0      0 ::1:25                  :::*                    LISTEN      0          33025      1969/master
tcp6       0      0 :::514                  :::*                    LISTEN      0          17607      1103/rsyslogd
tcp6       0      0 :::12900                :::*                    LISTEN      984        1364631    8392/java
udp        0      0 0.0.0.0:34051           0.0.0.0:*                           70         13488      1180/avahi-daemon:
udp        0      0 192.168.122.1:53        0.0.0.0:*                           0          10817      1988/dnsmasq
udp        0      0 0.0.0.0:67              0.0.0.0:*                           0          10814      1988/dnsmasq
udp        0      0 127.0.0.1:323           0.0.0.0:*                           0          32888      1141/chronyd
udp        0      0 0.0.0.0:514             0.0.0.0:*                           0          17602      1103/rsyslogd
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           70         13487      1180/avahi-daemon:
udp6       0      0 ::1:323                 :::*                                0          32889      1141/chronyd
udp6       0      0 :::514                  :::*                                0          17603      1103/rsyslogd
graylog>>/root>>
graylog>>/root>>
graylog>>/root>>
graylog>>/root>>sudo iptables -Ln
iptables: No chain/target/match by that name.
graylog>>/root>>sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  anywhere             anywhere             udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:bootps
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
INPUT_direct  all  --  anywhere             anywhere
INPUT_ZONES_SOURCE  all  --  anywhere             anywhere
INPUT_ZONES  all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere             ctstate INVALID
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     anywhere
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
FORWARD_direct  all  --  anywhere             anywhere
FORWARD_IN_ZONES_SOURCE  all  --  anywhere             anywhere
FORWARD_IN_ZONES  all  --  anywhere             anywhere
FORWARD_OUT_ZONES_SOURCE  all  --  anywhere             anywhere
FORWARD_OUT_ZONES  all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere             ctstate INVALID
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  anywhere             anywhere             udp dpt:bootpc
OUTPUT_direct  all  --  anywhere             anywhere

Chain FORWARD_IN_ZONES (1 references)
target     prot opt source               destination
FWDI_public  all  --  anywhere             anywhere            [goto]
FWDI_public  all  --  anywhere             anywhere            [goto]

Chain FORWARD_IN_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain FORWARD_OUT_ZONES (1 references)
target     prot opt source               destination
FWDO_public  all  --  anywhere             anywhere            [goto]
FWDO_public  all  --  anywhere             anywhere            [goto]

Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain FORWARD_direct (1 references)
target     prot opt source               destination

Chain FWDI_public (2 references)
target     prot opt source               destination
FWDI_public_log  all  --  anywhere             anywhere
FWDI_public_deny  all  --  anywhere             anywhere
FWDI_public_allow  all  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere

Chain FWDI_public_allow (1 references)
target     prot opt source               destination

Chain FWDI_public_deny (1 references)
target     prot opt source               destination

Chain FWDI_public_log (1 references)
target     prot opt source               destination

Chain FWDO_public (2 references)
target     prot opt source               destination
FWDO_public_log  all  --  anywhere             anywhere
FWDO_public_deny  all  --  anywhere             anywhere
FWDO_public_allow  all  --  anywhere             anywhere

Chain FWDO_public_allow (1 references)
target     prot opt source               destination

Chain FWDO_public_deny (1 references)
target     prot opt source               destination

Chain FWDO_public_log (1 references)
target     prot opt source               destination

Chain INPUT_ZONES (1 references)
target     prot opt source               destination
IN_public  all  --  anywhere             anywhere            [goto]
IN_public  all  --  anywhere             anywhere            [goto]

Chain INPUT_ZONES_SOURCE (1 references)
target     prot opt source               destination

Chain INPUT_direct (1 references)
target     prot opt source               destination

Chain IN_public (2 references)
target     prot opt source               destination
IN_public_log  all  --  anywhere             anywhere
IN_public_deny  all  --  anywhere             anywhere
IN_public_allow  all  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere

Chain IN_public_allow (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:cslistener ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:12900 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:commplex-main ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:bb ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:12201 ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:lxi-evntsvc ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:swxadmin ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:wap-wsp ctstate NEW
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ismserver ctstate NEW

Chain IN_public_deny (1 references)
target     prot opt source               destination

Chain IN_public_log (1 references)
target     prot opt source               destination

Chain OUTPUT_direct (1 references)
target     prot opt source               destination
4

You have to use the Logstash output (which is using the Beats/Lumberjack protocol), not the Elasticsearch output (which is supposed to send data to the Elasticsearch HTTP API).

5

Hi Jochen,
Thanks for clarifying and I have updated it under the Logstash output. I was able to successfully start the metricbeat service on the client machine. However I do not see any messages coming into the Beats Input I created in Graylog. Where do I look to see the metrics that should be coming in?

6

They should appear on the “All messages” stream (or alternatively the Universal Search).

Also make sure to check the logs of your Graylog node.
:arrow_right: http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

7

Hi,thanks, and my apologies, the metricbeat service on the client machine actually did not start successfully though it showed a ‘running’ status. Here are snapshots of the event log on the windows client, a snapshot of the currently configured logstash output, and snapshot of errors showing in the /var/log/graylog-server/server.log. I’ve researched the java ‘Unknown beats protocol version’ error but not clear on a solution.

image

image

java.lang.Exception: Unknown beats protocol version: 71
at org.graylog.plugins.beats.BeatsFrameDecoder.checkVersion(BeatsFrameDecoder.java:163) ~[?:?]
at org.graylog.plugins.beats.BeatsFrameDecoder.decode(BeatsFrameDecoder.java:92) ~[?:?]
at org.graylog.plugins.beats.BeatsFrameDecoder.decode(BeatsFrameDecoder.java:49) ~[?:?]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) ~[graylog.jar:?]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) ~[graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.handler.traffic.AbstractTrafficShapingHandler.messageReceived(AbstractTrafficShapingHandler.java:718) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124) [graylog.jar:?]
at org.graylog2.plugin.inputs.util.PacketInformationDumper.messageReceived(PacketInformationDumper.java:51) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [graylog.jar:?]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [graylog.jar:?]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [graylog.jar:?]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [graylog.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [graylog.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [graylog.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [graylog.jar:?]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [graylog.jar:?]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [graylog.jar:?]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2018-05-15T10:56:13.673-07:00 ERROR [NettyTransport] Error in Input [Beats/5af1e510cb28d97661ae994c] (channel [id: 0xc92936b1, /162.70.26.89:40910 :> /162.70.27.206:5043])
java.lang.Exception: Unknown beats protocol version: 69
at org.graylog.plugins.beats.BeatsFrameDecoder.checkVersion(BeatsFrameDecoder.java:163) ~[?:?]
at org.graylog.plugins.beats.BeatsFrameDecoder.decode(BeatsFrameDecoder.java:92) ~[?:?]
at org.graylog.plugins.beats.BeatsFrameDecoder.decode(BeatsFrameDecoder.java:49) ~[?:?]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) ~[graylog.jar:?]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.cleanup(ReplayingDecoder.java:554) ~[graylog.jar:?]
at org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:365) ~[graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) ~[graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.channelDisconnected(SimpleChannelHandler.java:199) [graylog.jar:?]
at org.graylog2.plugin.inputs.util.ConnectionCounter.channelDisconnected(ConnectionCounter.java:49) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:120) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.channelDisconnected(SimpleChannelHandler.java:199) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:120) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:208) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [graylog.jar:?]
at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) [graylog.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) [graylog.jar:?]
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:81) [graylog.jar:?]
at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.closeRequested(SimpleChannelHandler.java:334) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:260) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.closeRequested(SimpleChannelHandler.java:334) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:260) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.closeRequested(SimpleChannelHandler.java:334) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.handleDownstream(SimpleChannelHandler.java:260) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) [graylog.jar:?]
at org.jboss.netty.channel.Channels.close(Channels.java:812) [graylog.jar:?]
at org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:205) [graylog.jar:?]
at org.graylog2.plugin.inputs.transports.NettyTransport$RawMessageHandler.exceptionCaught(NettyTransport.java:353) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:130) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.exceptionCaught(SimpleChannelHandler.java:156) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:130) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.exceptionCaught(SimpleChannelHandler.java:156) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:130) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:153) [graylog.jar:?]
at org.graylog2.plugin.inputs.transports.NettyTransport$2$1.exceptionCaught(NettyTransport.java:210) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [graylog.jar:?]
at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:525) [graylog.jar:?]
at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:566) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.handler.traffic.AbstractTrafficShapingHandler.messageReceived(AbstractTrafficShapingHandler.java:718) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124) [graylog.jar:?]
at org.graylog2.plugin.inputs.util.PacketInformationDumper.messageReceived(PacketInformationDumper.java:51) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:124) [graylog.jar:?]
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [graylog.jar:?]
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [graylog.jar:?]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [graylog.jar:?]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [graylog.jar:?]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [graylog.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [graylog.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [graylog.jar:?]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [graylog.jar:?]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [graylog.jar:?]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [graylog.jar:?]
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
graylog>>/var/log/graylog-server>>

8

The configuration of Metricbeat is still wrong. For example, you’re still using the Elasticsearch output.
A line starting with ‘#’ is commented, i. e. it’s ignored.

Please refer to https://www.elastic.co/guide/en/beats/metricbeat/6.2/index.html for information how to configure Metricbeat.

9

Argh, sorry, that was a simple oversight. I’ve now commented out the elastic output and kept the logstash ouput uncommented. For some reason the windows service still will not start. I have read the guide you referenced above and will continue to work on this. I would appreciate if anyone in the graylog community has an example of a working metricbeat.yml using the logstash output. I am not having much luck locating a working example online. Thanks for all your help!

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.