Metricbeats Data Not Showing In Graylog

Graylog Central (peer support)
1

Hi, we are trying to send metric beats information into Graylog. I have installed metricbeat on a Linux machine and have configured it to point to our Graylog server. Input is created in Graylog. Firewall rule is in place on Graylog server to allow traffic. Able to telnet from source server to Graylog server on specified port. Metricbeat service appears to start successfully on source server, no discernable errors in metricbeat.log, just looks like its sending data. When I hit ‘show messages’ on the Input in Graylog it shows no messages. Setting up an extractor yields the same thing. Not sure where to look to find the metrics data. I have complete configurations saved in zip file for your review but can’t post it. Please let me know how to post the zip file.

2

Check the logs of your Graylog and Elasticsearch nodes.
:arrow_right: http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

You can post larger text files by using a pastebin service such as https://gist.github.com or https://0bin.net/ and share the link.

3

The pastebins only seem to work for .txt files. I thought I saw someone attach a zip file to a Graylog Topic. How did they accomplish that?

4

They didn’t.

Only image files (GIF, JPEG, PNG) are allowed up to a size of 3 MB.

Maybe you’re mixing that up with GitHub issues.

5

Ok fair enough. Here are the links to 2 gists, that contain a total of 5 txt files for troubleshooting. Also to this reply I will attach 3 .JPGs.
part 1 of 2

Part 2 of 2


15043%20beats%20input
15043 beats input.JPG1717×1296 163 KB
15043%20show%20received%20messages
15043 show received messages.JPG1716×637 71.1 KB
15043%20extractor
15043 extractor.JPG1712×1239 100 KB

6 
[2018-06-12 08:52:56,219][ERROR][index.engine             ] [graylog] [graylog_1][0] failed to merge
java.io.IOException: No space left on device

From here: Files for Graylog / Metricbeat Troubleshooting Part 1 of 2 · GitHub

7

Thanks for the update. That error was related to an older issue where /var had filled up and had no more disk space. That has since been rectified and the index rotation in graylog has been tweaked to rotate by size rather than number of messages and /var currently has 10+GB free. I should not have posted the elasticsearch log since its latest entry is much older than the most recent attempt to send metricbeat data into graylog.

8

Is there any load-balancer or reverse proxy between your Beats and the Graylog inputs?

2018-06-12T14:14:21.640-07:00 ERROR [NettyTransport] Error in Input [Beats/5b203386cb28d91ab691cfc8] (channel [id: 0xabdace1e, /162.70.26.89:58090 :> /162.70.27.206:15043])
java.lang.Exception: Unknown beats protocol version: 69
9

No these are simply two machines that live in the same domain and reside on the same subnet. There are no load balancer or reverse proxy between them. Can you confirm that when data does come in, I should see it under System > Inputs > Beats Input > Show Received Messages?

10

FYI: Easiest way to see if an Input is receiving anything at all is the stats part on the right of an input settings field in the UI.

Greetings,
Philipp

11

Hi Philipp, thanks for the reply! I do see some small activity in the status part next to the input settings but its not a lot. I am just confused as to where I will see the actual metrics of the machine that is sending in data, like cpu disk etc… I haven’t even been able to google an image of it online at this point. So I am not sure if it would show under received messages or if it would show elsewhere like for the sidecar-collectors under System > Collectors. Ive attached screenshots of the Input status and one of our current sidecar collectors which shows some slight system metrics.

collector%20sidecar
collector sidecar.JPG1685×1284 172 KB
beats%20stats
beats stats.JPG1682×407 42.8 KB

Thanks to all for your continued help on this.

12

Hi guys, any updates here? And can someone please confirm where I would expect to see the metricbeat data once it does start coming in?

13

The data points would appear as normal messages in Graylog.

14

Thank you, I appreciate the clarification. Is there anything else I can provide to help figure out the issue?

15

You can try running Metricbeat in debug mode or with verbose logging to see what it’s doing exactly.

https://www.elastic.co/guide/en/beats/metricbeat/6.3/enable-metricbeat-debugging.html
https://www.elastic.co/guide/en/beats/metricbeat/6.3/configuration-logging.html

16

Ok I am now running metricbeat in full debug mode. Please see the GitHubGist pasted below for a sample of the metricbeat log output. It appears to be sending in data and I do see some data in Beats Input > Show Metrics, but no data points are showing up in Beats Input > Show Received Messages.

input
input.JPG1709×535 69.8 KB
Input%20Metrics
Input Metrics.JPG1714×1028 131 KB

17

I’m unable to reproduce the issue with Graylog 2.4.5 (Docker), Elasticsearch 5.6.10 (Docker), and Metricbeat 6.2.4 (which seems to be what you’re using).

Here’s all the information you would need to reproduce:

Beats input configuration

bind_address: 0.0.0.0
override_source: <empty>
port: 5044
recv_buffer_size: 1048576
tcp_keepalive: false
tls_cert_file: <empty>
tls_client_auth: disabled
tls_client_auth_cert_file: <empty>
tls_enable: false
tls_key_file: <empty>
tls_key_password: ********

29
Bildschirmfoto 2018-06-19 um 22.00.29.png990×314 22.7 KB

Metricbeat configuration

#==========================  Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

metricbeat.modules:

#------------------------------- System Module -------------------------------
- module: system
  metricsets:
    - cpu             # CPU usage
    - load            # CPU load averages
    - memory          # Memory usage
    - uptime          # System Uptime
  enabled: true
  period: 10s
  processes: ['.*']

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

#================================ Outputs =====================================
#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

Metricbeat configuration test

$ ./metricbeat test config
Config OK
$ ./metricbeat test output
logstash: localhost:5044...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: ::1, 127.0.0.1
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK

Metricbeat logs

$ ./metricbeat -v -e
2018-06-19T22:00:03.171+0200	INFO	instance/beat.go:468	Home path: [/Users/joschi/Downloads/metricbeat-6.2.4-darwin-x86_64] Config path: [/Users/joschi/Downloads/metricbeat-6.2.4-darwin-x86_64] Data path: [/Users/joschi/Downloads/metricbeat-6.2.4-darwin-x86_64/data] Logs path: [/Users/joschi/Downloads/metricbeat-6.2.4-darwin-x86_64/logs]
2018-06-19T22:00:03.171+0200	INFO	instance/beat.go:475	Beat UUID: 7a06eaf6-98b0-47f4-9899-5f5f0966baf0
2018-06-19T22:00:03.171+0200	INFO	instance/beat.go:213	Setup Beat: metricbeat; Version: 6.2.4
2018-06-19T22:00:03.172+0200	INFO	pipeline/module.go:76	Beat name: joschi-mbp15-lan.lan
2018-06-19T22:00:03.172+0200	INFO	instance/beat.go:301	metricbeat start running.
2018-06-19T22:00:03.173+0200	INFO	[monitoring]	log/log.go:97	Starting metrics logging every 30s
2018-06-19T22:00:03.175+0200	INFO	cfgfile/reload.go:127	Config reloader started
2018-06-19T22:00:03.177+0200	INFO	cfgfile/reload.go:258	Starting 3 runners ...
2018-06-19T22:00:03.177+0200	INFO	cfgfile/reload.go:219	Loading of config files completed.
2018-06-19T22:00:33.178+0200	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":110,"time":110},"total":{"ticks":532,"time":532,"value":532},"user":{"ticks":422,"time":422}},"info":{"ephemeral_id":"b33e4d33-7901-47af-937a-c6e82f98da88","uptime":{"ms":30026}},"memstats":{"gc_next":7506448,"memory_alloc":3771832,"memory_total":408141320,"rss":30531584}},"libbeat":{"config":{"module":{"running":3,"starts":3},"reloads":1},"output":{"events":{"acked":95,"batches":9,"total":95},"read":{"bytes":54},"type":"logstash","write":{"bytes":14242}},"pipeline":{"clients":7,"events":{"active":3,"filtered":1,"published":98,"retry":38,"total":99},"queue":{"acked":95}}},"metricbeat":{"system":{"cpu":{"events":7,"success":7},"filesystem":{"events":9,"success":9},"fsstat":{"events":1,"success":1},"load":{"events":7,"success":7},"memory":{"events":7,"success":7},"network":{"events":39,"success":39},"process":{"events":22,"success":22},"process_summary":{"events":3,"success":3},"uptime":{"events":4,"success":4}}},"system":{"cpu":{"cores":8},"load":{"1":2.5127,"15":2.0269,"5":2.2207,"norm":{"1":0.3141,"15":0.2534,"5":0.2776}}}}}}
2018-06-19T22:01:03.173+0200	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":175,"time":175},"total":{"ticks":853,"time":853,"value":853},"user":{"ticks":678,"time":678}},"info":{"ephemeral_id":"b33e4d33-7901-47af-937a-c6e82f98da88","uptime":{"ms":60024}},"memstats":{"gc_next":7600368,"memory_alloc":5312760,"memory_total":662848344,"rss":143360}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":83,"batches":9,"total":83},"read":{"bytes":54},"write":{"bytes":12942}},"pipeline":{"clients":7,"events":{"active":0,"published":80,"total":80},"queue":{"acked":83}}},"metricbeat":{"system":{"cpu":{"events":5,"success":5},"load":{"events":5,"success":5},"memory":{"events":5,"success":5},"network":{"events":39,"success":39},"process":{"events":20,"success":20},"process_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"system":{"load":{"1":2.2461,"15":2.0181,"5":2.1826,"norm":{"1":0.2808,"15":0.2523,"5":0.2728}}}}}}
2018-06-19T22:01:33.170+0200	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":247,"time":247},"total":{"ticks":1213,"time":1213,"value":1213},"user":{"ticks":966,"time":966}},"info":{"ephemeral_id":"b33e4d33-7901-47af-937a-c6e82f98da88","uptime":{"ms":90022}},"memstats":{"gc_next":7538416,"memory_alloc":3775440,"memory_total":915796616,"rss":81920}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":90,"batches":9,"total":90},"read":{"bytes":54},"write":{"bytes":13618}},"pipeline":{"clients":7,"events":{"active":0,"filtered":1,"published":90,"total":91},"queue":{"acked":90}}},"metricbeat":{"system":{"cpu":{"events":6,"success":6},"filesystem":{"events":9,"success":9},"fsstat":{"events":1,"success":1},"load":{"events":6,"success":6},"memory":{"events":6,"success":6},"network":{"events":39,"success":39},"process":{"events":18,"success":18},"process_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"system":{"load":{"1":2.4492,"15":2.0386,"5":2.2295,"norm":{"1":0.3062,"15":0.2548,"5":0.2787}}}}}}
2018-06-19T22:02:03.170+0200	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":316,"time":316},"total":{"ticks":1568,"time":1568,"value":1568},"user":{"ticks":1252,"time":1252}},"info":{"ephemeral_id":"b33e4d33-7901-47af-937a-c6e82f98da88","uptime":{"ms":120024}},"memstats":{"gc_next":7540912,"memory_alloc":3776736,"memory_total":1167849264,"rss":32768}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":81,"batches":9,"total":81},"read":{"bytes":54},"write":{"bytes":12442}},"pipeline":{"clients":7,"events":{"active":0,"published":81,"total":81},"queue":{"acked":81}}},"metricbeat":{"system":{"cpu":{"events":6,"success":6},"load":{"events":6,"success":6},"memory":{"events":6,"success":6},"network":{"events":39,"success":39},"process":{"events":18,"success":18},"process_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"system":{"load":{"1":2.4258,"15":2.0532,"5":2.252,"norm":{"1":0.3032,"15":0.2567,"5":0.2815}}}}}}
2018-06-19T22:02:33.171+0200	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":388,"time":388},"total":{"ticks":1909,"time":1909,"value":1909},"user":{"ticks":1521,"time":1521}},"info":{"ephemeral_id":"b33e4d33-7901-47af-937a-c6e82f98da88","uptime":{"ms":150025}},"memstats":{"gc_next":7537856,"memory_alloc":3782432,"memory_total":1419999040,"rss":40960}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":90,"batches":9,"total":90},"read":{"bytes":54},"write":{"bytes":13602}},"pipeline":{"clients":7,"events":{"active":0,"filtered":1,"published":90,"total":91},"queue":{"acked":90}}},"metricbeat":{"system":{"cpu":{"events":6,"success":6},"filesystem":{"events":9,"success":9},"fsstat":{"events":1,"success":1},"load":{"events":6,"success":6},"memory":{"events":6,"success":6},"network":{"events":39,"success":39},"process":{"events":18,"success":18},"process_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"system":{"load":{"1":2.4741,"15":2.0679,"5":2.2754,"norm":{"1":0.3093,"15":0.2585,"5":0.2844}}}}}}
2018-06-19T22:03:03.167+0200	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":458,"time":458},"total":{"ticks":2248,"time":2248,"value":2248},"user":{"ticks":1790,"time":1790}},"info":{"ephemeral_id":"b33e4d33-7901-47af-937a-c6e82f98da88","uptime":{"ms":180023}},"memstats":{"gc_next":7619056,"memory_alloc":3825568,"memory_total":1672069208,"rss":16384}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":82,"batches":9,"total":82},"read":{"bytes":54},"write":{"bytes":12723}},"pipeline":{"clients":7,"events":{"active":0,"published":82,"total":82},"queue":{"acked":82}}},"metricbeat":{"system":{"cpu":{"events":6,"success":6},"load":{"events":6,"success":6},"memory":{"events":6,"success":6},"network":{"events":39,"success":39},"process":{"events":19,"success":19},"process_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"system":{"load":{"1":2.2744,"15":2.0649,"5":2.2466,"norm":{"1":0.2843,"15":0.2581,"5":0.2808}}}}}}
2018-06-19T22:03:33.169+0200	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":527,"time":527},"total":{"ticks":2587,"time":2587,"value":2587},"user":{"ticks":2060,"time":2060}},"info":{"ephemeral_id":"b33e4d33-7901-47af-937a-c6e82f98da88","uptime":{"ms":210027}},"memstats":{"gc_next":7625424,"memory_alloc":5331944,"memory_total":1924462480,"rss":28672}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":90,"batches":9,"total":90},"read":{"bytes":54},"write":{"bytes":13910}},"pipeline":{"clients":7,"events":{"active":0,"filtered":1,"published":90,"total":91},"queue":{"acked":90}}},"metricbeat":{"system":{"cpu":{"events":6,"success":6},"filesystem":{"events":9,"success":9},"fsstat":{"events":1,"success":1},"load":{"events":6,"success":6},"memory":{"events":6,"success":6},"network":{"events":39,"success":39},"process":{"events":18,"success":18},"process_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"system":{"load":{"1":2.1597,"15":2.062,"5":2.2212,"norm":{"1":0.27,"15":0.2578,"5":0.2776}}}}}}
2018-06-19T22:04:03.165+0200	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":597,"time":597},"total":{"ticks":2921,"time":2921,"value":2921},"user":{"ticks":2324,"time":2324}},"info":{"ephemeral_id":"b33e4d33-7901-47af-937a-c6e82f98da88","uptime":{"ms":240023}},"memstats":{"gc_next":7560048,"memory_alloc":5294856,"memory_total":2175712000,"rss":28672}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":81,"batches":9,"total":81},"read":{"bytes":54},"write":{"bytes":12603}},"pipeline":{"clients":7,"events":{"active":0,"published":81,"total":81},"queue":{"acked":81}}},"metricbeat":{"system":{"cpu":{"events":6,"success":6},"load":{"events":6,"success":6},"memory":{"events":6,"success":6},"network":{"events":39,"success":39},"process":{"events":18,"success":18},"process_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"system":{"load":{"1":1.9331,"15":2.0479,"5":2.1685,"norm":{"1":0.2416,"15":0.256,"5":0.2711}}}}}}
2018-06-19T22:04:33.165+0200	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":664,"time":664},"total":{"ticks":3260,"time":3260,"value":3260},"user":{"ticks":2596,"time":2596}},"info":{"ephemeral_id":"b33e4d33-7901-47af-937a-c6e82f98da88","uptime":{"ms":270024}},"memstats":{"gc_next":7623024,"memory_alloc":3818032,"memory_total":2427066584,"rss":20480}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":90,"batches":9,"total":90},"read":{"bytes":54},"write":{"bytes":13570}},"pipeline":{"clients":7,"events":{"active":0,"filtered":1,"published":90,"total":91},"queue":{"acked":90}}},"metricbeat":{"system":{"cpu":{"events":6,"success":6},"filesystem":{"events":9,"success":9},"fsstat":{"events":1,"success":1},"load":{"events":6,"success":6},"memory":{"events":6,"success":6},"network":{"events":39,"success":39},"process":{"events":18,"success":18},"process_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"system":{"load":{"1":2.3267,"15":2.0801,"5":2.2451,"norm":{"1":0.2908,"15":0.26,"5":0.2806}}}}}}
2018-06-19T22:05:03.162+0200	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":738,"time":738},"total":{"ticks":3606,"time":3606,"value":3606},"user":{"ticks":2868,"time":2868}},"info":{"ephemeral_id":"b33e4d33-7901-47af-937a-c6e82f98da88","uptime":{"ms":300023}},"memstats":{"gc_next":7567552,"memory_alloc":3792352,"memory_total":2681510624,"rss":24576}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":82,"batches":9,"total":82},"read":{"bytes":54},"write":{"bytes":12762}},"pipeline":{"clients":7,"events":{"active":0,"published":82,"total":82},"queue":{"acked":82}}},"metricbeat":{"system":{"cpu":{"events":6,"success":6},"load":{"events":6,"success":6},"memory":{"events":6,"success":6},"network":{"events":39,"success":39},"process":{"events":19,"success":19},"process_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"system":{"load":{"1":2.4971,"15":2.0996,"5":2.2861,"norm":{"1":0.3121,"15":0.2625,"5":0.2858}}}}}}
2018-06-19T22:05:33.162+0200	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":808,"time":808},"total":{"ticks":3942,"time":3942,"value":3942},"user":{"ticks":3134,"time":3134}},"info":{"ephemeral_id":"b33e4d33-7901-47af-937a-c6e82f98da88","uptime":{"ms":330024}},"memstats":{"gc_next":7577328,"memory_alloc":3803840,"memory_total":2941614416,"rss":-1552384}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":94,"batches":9,"total":94},"read":{"bytes":54},"write":{"bytes":14420}},"pipeline":{"clients":7,"events":{"active":0,"filtered":1,"published":94,"total":95},"queue":{"acked":94}}},"metricbeat":{"system":{"cpu":{"events":6,"success":6},"filesystem":{"events":9,"success":9},"fsstat":{"events":1,"success":1},"load":{"events":6,"success":6},"memory":{"events":6,"success":6},"network":{"events":39,"success":39},"process":{"events":22,"success":22},"process_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"system":{"load":{"1":2.8877,"15":2.1465,"5":2.4009,"norm":{"1":0.361,"15":0.2683,"5":0.3001}}}}}}
2018-06-19T22:06:03.159+0200	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":874,"time":874},"total":{"ticks":4275,"time":4275,"value":4275},"user":{"ticks":3401,"time":3401}},"info":{"ephemeral_id":"b33e4d33-7901-47af-937a-c6e82f98da88","uptime":{"ms":360022}},"memstats":{"gc_next":7577520,"memory_alloc":4065024,"memory_total":3198443952,"rss":12288}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":84,"batches":9,"total":84},"read":{"bytes":54},"write":{"bytes":13151}},"pipeline":{"clients":7,"events":{"active":0,"published":84,"total":84},"queue":{"acked":84}}},"metricbeat":{"system":{"cpu":{"events":6,"success":6},"load":{"events":6,"success":6},"memory":{"events":6,"success":6},"network":{"events":39,"success":39},"process":{"events":21,"success":21},"process_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"system":{"load":{"1":2.269,"15":2.1172,"5":2.2974,"norm":{"1":0.2836,"15":0.2646,"5":0.2872}}}}}}
2018-06-19T22:06:33.161+0200	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":941,"time":941},"total":{"ticks":4622,"time":4622,"value":4622},"user":{"ticks":3681,"time":3681}},"info":{"ephemeral_id":"b33e4d33-7901-47af-937a-c6e82f98da88","uptime":{"ms":390025}},"memstats":{"gc_next":7647312,"memory_alloc":5338840,"memory_total":3451945912,"rss":12288}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":95,"batches":9,"total":95},"read":{"bytes":54},"write":{"bytes":14493}},"pipeline":{"clients":7,"events":{"active":0,"filtered":1,"published":95,"total":96},"queue":{"acked":95}}},"metricbeat":{"system":{"cpu":{"events":6,"success":6},"filesystem":{"events":9,"success":9},"fsstat":{"events":1,"success":1},"load":{"events":6,"success":6},"memory":{"events":6,"success":6},"network":{"events":39,"success":39},"process":{"events":23,"success":23},"process_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"system":{"load":{"1":2.3394,"15":2.1289,"5":2.3145,"norm":{"1":0.2924,"15":0.2661,"5":0.2893}}}}}}
2018-06-19T22:07:03.161+0200	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1013,"time":1013},"total":{"ticks":4973,"time":4973,"value":4973},"user":{"ticks":3960,"time":3960}},"info":{"ephemeral_id":"b33e4d33-7901-47af-937a-c6e82f98da88","uptime":{"ms":420026}},"memstats":{"gc_next":7581648,"memory_alloc":3797264,"memory_total":3704808240,"rss":20480}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":84,"batches":9,"total":84},"read":{"bytes":54},"write":{"bytes":13210}},"pipeline":{"clients":7,"events":{"active":0,"published":84,"total":84},"queue":{"acked":84}}},"metricbeat":{"system":{"cpu":{"events":7,"success":7},"load":{"events":7,"success":7},"memory":{"events":6,"success":6},"network":{"events":39,"success":39},"process":{"events":21,"success":21},"process_summary":{"events":3,"success":3},"uptime":{"events":3,"success":3}}},"system":{"load":{"1":2.0508,"15":2.1055,"5":2.2373,"norm":{"1":0.2563,"15":0.2632,"5":0.2797}}}}}}
^C2018-06-19T22:07:04.155+0200	INFO	cfgfile/reload.go:222	Dynamic config reloader stopped
2018-06-19T22:07:04.155+0200	INFO	instance/beat.go:308	metricbeat stopped.
2018-06-19T22:07:04.156+0200	INFO	[monitoring]	log/log.go:132	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1034,"time":1034},"total":{"ticks":5069,"time":5069,"value":5069},"user":{"ticks":4035,"time":4035}},"info":{"ephemeral_id":"b33e4d33-7901-47af-937a-c6e82f98da88","uptime":{"ms":421021}},"memstats":{"gc_next":5718256,"memory_alloc":3248752,"memory_total":3785868328,"rss":29483008}},"libbeat":{"config":{"module":{"running":3,"starts":3},"reloads":1},"output":{"events":{"acked":1221,"batches":126,"total":1221},"read":{"bytes":756},"type":"logstash","write":{"bytes":187688}},"pipeline":{"clients":6,"events":{"active":36,"filtered":8,"published":1257,"retry":38,"total":1265},"queue":{"acked":1221}}},"metricbeat":{"system":{"cpu":{"events":86,"success":86},"filesystem":{"events":72,"success":72},"fsstat":{"events":8,"success":8},"load":{"events":85,"success":85},"memory":{"events":85,"success":85},"network":{"events":559,"success":559},"process":{"events":283,"success":283},"process_summary":{"events":43,"success":43},"uptime":{"events":44,"success":44}}},"system":{"cpu":{"cores":8},"load":{"1":2.0508,"15":2.1055,"5":2.2373,"norm":{"1":0.2563,"15":0.2632,"5":0.2797}}}}}}
2018-06-19T22:07:04.156+0200	INFO	[monitoring]	log/log.go:133	Uptime: 7m1.022626497s
2018-06-19T22:07:04.156+0200	INFO	[monitoring]	log/log.go:110	Stopping metrics logging.

Example messages in Graylog

49
Bildschirmfoto 2018-06-19 um 22.04.49.png1517×1343 290 KB

docker-compose.yml

version: '2'
services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongo:
    image: mongo:3
  # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.10
    environment:
      - http.host=0.0.0.0
      - discovery.type=single-node
      # Disable X-Pack security: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/security-settings.html#general-security-settings
      - xpack.security.enabled=false
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:2.4.5-1
    environment:
      # CHANGE ME!
      - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      - GRAYLOG_WEB_ENDPOINT_URI=http://127.0.0.1:9000/api
      - GRAYLOG_MESSAGE_JOURNAL_MAX_SIZE=10gb
    links:
      - mongo
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Beats input
      - 5044:5044
1 Like
18

Thanks for attempting to reproduce the issue. I will take this info and work with my team. I really appreciate your time.

19

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.