Graylog recieves messages but don't show them

Graylog Central (peer support)
1

Hey, I run filebeat logstash and graylog2 on docker.
This is the output in my logstash.conf:

output {
	gelf{
		host => "graylog_1"
		port => 5555
	}
}

I created an input in graylog with gelf udp and also tried raw udp. Graylogs reciews logs, it shows arround 70 messages per second. But when i search for them, even in the future because i read about those timezone problem, there are no messages.
This is my docker compose to start Graylog:

version: '2'

services:
    mongodb:
        image: mongo:3
        volumes:
            - mongo_data:/mongo/db
        ports:
            - "28017:28017"
    elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:5.6.3
        volumes:
            - es_data:/usr/share/elasticsearch/data
        environment:
            - http.host=0.0.0.0
            - transport.host=localhost
            - network.host=0.0.0.0
            - xpack.security.enabled=false
            - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
        ulimits:
            memlock:
                soft: -1
                hard: -1
        mem_limit: 1g
    graylog:
        image: graylog/graylog:latest
        volumes:
            - graylog_journal:/usr/share/graylog/data/journal
        environment:
            GRAYLOG_PASSWORD_SECRET: somepasswordpepper
            GRAYLOG_ROOT_PASSWORD_SHA2: 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
            GRAYLOG_WEB_ENDPOINT_URI: http://127.0.0.1:9000/api
        links:
            - mongodb:mongo
            - elasticsearch
        depends_on:
            - mongodb
            - elasticsearch
        ports:
            - "9000:9000"
            - "5555:5555/udp"
            - "5444:5444"
            - "12201:12201/udp"
    logstash:
        image: iot-logstash:latest
        ports:
            - "127.0.0.1:5044:5044"
        depends_on:
            - graylog 
volumes:
    mongo_data:
        driver: local
    es_data:
        driver: local
    graylog_journal:
        driver: local

I am not really deep into docker or graylog. So I don’t know whether I did something wrong, but I can’t find a solution for this problem. Does someone know what the problem is?

Thank you,
Lukas

2

What’s in the logs of your Graylog and Elasticsearch nodes?

3

Elasticsearch:

[2018-03-04T15:59:42,874][INFO ][o.e.n.Node               ] [] initializing ...
[2018-03-04T15:59:42,976][INFO ][o.e.e.NodeEnvironment    ] [pXBkbjm] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda1)]], net usable_space [36.2gb], net total_space [58.8gb], spins? [possibly], types [ext4]
[2018-03-04T15:59:42,976][INFO ][o.e.e.NodeEnvironment    ] [pXBkbjm] heap size [494.9mb], compressed ordinary object pointers [true]
[2018-03-04T15:59:43,025][INFO ][o.e.n.Node               ] node name [pXBkbjm] derived from node ID [pXBkbjm3SVmBt1PUAxRNPw]; set [node.name] to override
[2018-03-04T15:59:43,025][INFO ][o.e.n.Node               ] version[5.6.3], pid[1], build[1a2f265/2017-10-06T20:33:39.012Z], OS[Linux/4.9.60-linuxkit-aufs/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
[2018-03-04T15:59:43,026][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch]
[2018-03-04T15:59:44,458][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [aggs-matrix-stats]
[2018-03-04T15:59:44,459][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [ingest-common]
[2018-03-04T15:59:44,459][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [lang-expression]
[2018-03-04T15:59:44,459][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [lang-groovy]
[2018-03-04T15:59:44,459][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [lang-mustache]
[2018-03-04T15:59:44,459][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [lang-painless]
[2018-03-04T15:59:44,459][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [parent-join]
[2018-03-04T15:59:44,459][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [percolator]
[2018-03-04T15:59:44,459][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [reindex]
[2018-03-04T15:59:44,459][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [transport-netty3]
[2018-03-04T15:59:44,459][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [transport-netty4]
[2018-03-04T15:59:44,460][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded plugin [ingest-geoip]
[2018-03-04T15:59:44,460][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded plugin [ingest-user-agent]
[2018-03-04T15:59:44,460][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded plugin [x-pack]
[2018-03-04T15:59:47,169][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/55] [Main.cc@128] controller (64 bit): Version 5.6.3 (Build 5bf431d29b5bff) Copyright (c) 2017 Elasticsearch BV
[2018-03-04T15:59:47,289][INFO ][o.e.d.DiscoveryModule    ] [pXBkbjm] using discovery type [zen]
[2018-03-04T15:59:48,422][INFO ][o.e.n.Node               ] initialized
[2018-03-04T15:59:48,422][INFO ][o.e.n.Node               ] [pXBkbjm] starting ...
[2018-03-04T15:59:48,671][INFO ][o.e.t.TransportService   ] [pXBkbjm] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2018-03-04T15:59:51,743][INFO ][o.e.c.s.ClusterService   ] [pXBkbjm] new_master {pXBkbjm}{pXBkbjm3SVmBt1PUAxRNPw}{GqiqXQmHQ4GNA8uwjqAiRw}{localhost}{127.0.0.1:9300}{ml.max_open_jobs=10, ml.enabled=true}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-03-04T15:59:51,793][INFO ][o.e.h.n.Netty4HttpServerTransport] [pXBkbjm] publish_address {172.19.0.2:9200}, bound_addresses {0.0.0.0:9200}
[2018-03-04T15:59:51,793][INFO ][o.e.n.Node               ] [pXBkbjm] started
[2018-03-04T15:59:52,645][INFO ][o.e.l.LicenseService     ] [pXBkbjm] license [047fca8f-9704-4c57-a734-1e368c714736] mode [trial] - valid
[2018-03-04T15:59:52,652][INFO ][o.e.g.GatewayService     ] [pXBkbjm] recovered [10] indices into cluster_state
[2018-03-04T15:59:56,257][INFO ][o.e.c.r.a.AllocationService] [pXBkbjm] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[graylog_0][2]] ...]).
[2018-03-06T07:53:28,502][INFO ][o.e.n.Node               ] [] initializing ...
[2018-03-06T07:53:28,629][INFO ][o.e.e.NodeEnvironment    ] [pXBkbjm] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda1)]], net usable_space [36.3gb], net total_space [58.8gb], spins? [possibly], types [ext4]
[2018-03-06T07:53:28,630][INFO ][o.e.e.NodeEnvironment    ] [pXBkbjm] heap size [494.9mb], compressed ordinary object pointers [true]
[2018-03-06T07:53:28,669][INFO ][o.e.n.Node               ] node name [pXBkbjm] derived from node ID [pXBkbjm3SVmBt1PUAxRNPw]; set [node.name] to override
[2018-03-06T07:53:28,669][INFO ][o.e.n.Node               ] version[5.6.3], pid[1], build[1a2f265/2017-10-06T20:33:39.012Z], OS[Linux/4.9.60-linuxkit-aufs/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
[2018-03-06T07:53:28,669][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch]
[2018-03-06T07:53:29,996][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [aggs-matrix-stats]
[2018-03-06T07:53:29,996][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [ingest-common]
[2018-03-06T07:53:29,996][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [lang-expression]
[2018-03-06T07:53:29,996][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [lang-groovy]
[2018-03-06T07:53:29,996][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [lang-mustache]
[2018-03-06T07:53:29,996][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [lang-painless]
[2018-03-06T07:53:29,996][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [parent-join]
[2018-03-06T07:53:29,996][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [percolator]
[2018-03-06T07:53:29,996][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [reindex]
[2018-03-06T07:53:29,996][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [transport-netty3]
[2018-03-06T07:53:29,996][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded module [transport-netty4]
[2018-03-06T07:53:29,997][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded plugin [ingest-geoip]
[2018-03-06T07:53:29,997][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded plugin [ingest-user-agent]
[2018-03-06T07:53:29,997][INFO ][o.e.p.PluginsService     ] [pXBkbjm] loaded plugin [x-pack]
[2018-03-06T07:53:31,822][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/55] [Main.cc@128] controller (64 bit): Version 5.6.3 (Build 5bf431d29b5bff) Copyright (c) 2017 Elasticsearch BV
[2018-03-06T07:53:31,861][INFO ][o.e.d.DiscoveryModule    ] [pXBkbjm] using discovery type [zen]
[2018-03-06T07:53:32,546][INFO ][o.e.n.Node               ] initialized
[2018-03-06T07:53:32,546][INFO ][o.e.n.Node               ] [pXBkbjm] starting ...
[2018-03-06T07:53:32,754][INFO ][o.e.t.TransportService   ] [pXBkbjm] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2018-03-06T07:53:35,815][INFO ][o.e.c.s.ClusterService   ] [pXBkbjm] new_master {pXBkbjm}{pXBkbjm3SVmBt1PUAxRNPw}{6nPVMJYfQcewaNt2EqMShw}{localhost}{127.0.0.1:9300}{ml.max_open_jobs=10, ml.enabled=true}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-03-06T07:53:35,893][INFO ][o.e.h.n.Netty4HttpServerTransport] [pXBkbjm] publish_address {172.19.0.3:9200}, bound_addresses {0.0.0.0:9200}
[2018-03-06T07:53:35,894][INFO ][o.e.n.Node               ] [pXBkbjm] started
[2018-03-06T07:53:36,547][INFO ][o.e.l.LicenseService     ] [pXBkbjm] license [047fca8f-9704-4c57-a734-1e368c714736] mode [trial] - valid
[2018-03-06T07:53:36,550][WARN ][o.e.l.LicenseService     ] [pXBkbjm] 
[2018-03-06T07:53:36,554][INFO ][o.e.g.GatewayService     ] [pXBkbjm] recovered [10] indices into cluster_state
[2018-03-06T07:53:40,229][INFO ][o.e.c.r.a.AllocationService] [pXBkbjm] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.monitoring-es-6-2018.03.04][0]] ...]).
[2018-03-06T07:53:42,773][INFO ][o.e.c.m.MetaDataCreateIndexService] [pXBkbjm] [.monitoring-es-6-2018.03.06] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[1], mappings [doc]
[2018-03-06T07:54:38,913][INFO ][o.e.c.m.MetaDataCreateIndexService] [pXBkbjm] [.watcher-history-6-2018.03.06] creating index, cause [auto(bulk api)], templates [.watch-history-6], shards [1]/[1], mappings [doc]
[2018-03-06T07:54:39,014][INFO ][o.e.c.m.MetaDataMappingService] [pXBkbjm] [.watcher-history-6-2018.03.06/PcSQSuY1RH6ezZYEv8KW7g] update_mapping [doc]
[2018-03-06T07:54:39,028][INFO ][o.e.c.m.MetaDataMappingService] [pXBkbjm] [.watcher-history-6-2018.03.06/PcSQSuY1RH6ezZYEv8KW7g] update_mapping [doc]
4

There doesn’t seem to be a connection from Graylog whatsoever.

Make sure that the Elasticsearch connection is correctly configured in your Graylog configuration.

5

Ok got it! Thank you very much

6

How did u fix?
What’s exactly error, guy?

7

Sorry for the late response.
Ehm there was something wrong with the volumes… Elastic wasn’t able to write on the paths i think there was some probleme to the permissions…
I changed the volumes to something like:
volumes:
- $HOME/data/elastic:/data/elastic
and it worked fine

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.