Input receive messages but nothing in stream/search

Graylog Central (peer support)
1

Hello all,

I am a little new in graylog world.
I have installed Graylog2 (2.2.2) with Docker.

My installation was working well until no more space on disk.:astonished:
So i have increase disk space via LVM.
I killed my dockers and restarted them.
Inputs are still collecting logs but no messages are shown in search or in stream (only old messages before “no more space on disk”)

You can see my docker-compose.yml file

version: '2'
services:
  some-mongo:
image: "mongo:3"
volumes:
  - /graylog/data/mongo:/data/db
  some-elasticsearch:
image: "elasticsearch:2"
command: "elasticsearch -Des.cluster.name='graylog'"
volumes:
  - /graylog/data/elasticsearch:/usr/share/elasticsearch/data
  graylog:
image: graylog2/server
volumes:
  - /graylog/data/journal:/usr/share/graylog/data/journal
  - /graylog/config:/usr/share/graylog/data/config
links:
  - some-mongo:mongo
  - some-elasticsearch:elasticsearch
environment:
  GRAYLOG_PASSWORD_SECRET: ****
  GRAYLOG_ROOT_PASSWORD_SHA2: ****
  GRAYLOG_WEB_ENDPOINT_URI: http://x.x.x.x:9000/api
  GRAYLOG_ROOT_TIMEZONE: Europe/Paris
ports:
  - "9000:9000"
  - "12201/udp:12201/udp"
  - "1514/udp:1514/udp"
  - "1614/tcp:1614/tcp"
  - "5414/udp:5414/udp"

I don’t really now what is my problem.

Thank you for your help :slight_smile:

2

Please post the logs of your Elasticsearch and Graylog containers.

3

Hi Jochen,

Thank you for your reply.
There are logs below :slight_smile:

sudo docker logs config_some-elasticsearch_1

    [2017-04-10 14:39:06,197][WARN ][bootstrap                ] unable to install syscall filter: seccomp unavailable: your kernel is buggy and you should upgrade
    [2017-04-10 14:39:07,440][INFO ][node                     ] [Baron Strucker] version[2.4.4], pid[1], build[fcbb46d/2017-01-03T11:33:16Z]
    [2017-04-10 14:39:07,447][INFO ][node                     ] [Baron Strucker] initializing ...
    [2017-04-10 14:39:11,157][INFO ][plugins                  ] [Baron Strucker] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
    [2017-04-10 14:39:11,276][INFO ][env                      ] [Baron Strucker] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/SRVWEB03--vg-root)]], net usable_space [267.4mb], net total_space [10.7gb], spins? [possibly], types [ext4]
    [2017-04-10 14:39:11,277][INFO ][env                      ] [Baron Strucker] heap size [1015.6mb], compressed ordinary object pointers [true]
    [2017-04-10 14:39:25,744][INFO ][node                     ] [Baron Strucker] initialized
    [2017-04-10 14:39:25,752][INFO ][node                     ] [Baron Strucker] starting ...
    [2017-04-10 14:39:26,368][INFO ][transport                ] [Baron Strucker] publish_address {172.18.0.2:9300}, bound_addresses {[::]:9300}
    [2017-04-10 14:39:26,420][INFO ][discovery                ] [Baron Strucker] graylog/h2CcecesRl6AKhCg0ROcmw
    [2017-04-10 14:39:29,865][INFO ][cluster.service          ] [Baron Strucker] new_master {Baron Strucker}{h2CcecesRl6AKhCg0ROcmw}{172.18.0.2}{172.18.0.2:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
    [2017-04-10 14:39:29,941][INFO ][http                     ] [Baron Strucker] publish_address {172.18.0.2:9200}, bound_addresses {[::]:9200}
    [2017-04-10 14:39:29,947][INFO ][node                     ] [Baron Strucker] started
    [2017-04-10 14:39:30,367][INFO ][gateway                  ] [Baron Strucker] recovered [1] indices into cluster_state
    [2017-04-10 14:39:38,896][INFO ][cluster.routing.allocation] [Baron Strucker] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][3], [graylog_0][3], [graylog_0][2], [graylog_0][0], [graylog_0][0]] ...]).
    [2017-04-10 14:40:00,382][WARN ][cluster.routing.allocation.decider] [Baron Strucker] high disk watermark [90%] exceeded on [h2CcecesRl6AKhCg0ROcmw][Baron Strucker][/usr/share/elasticsearch/data/graylog/nodes/0] free: 267.2mb[2.4%], shards will be relocated away from this node
    [2017-04-10 14:40:00,386][INFO ][cluster.routing.allocation.decider] [Baron Strucker] rerouting shards: [high disk watermark exceeded on one or more nodes]
    [2017-04-10 14:40:12,650][INFO ][cluster.service          ] [Baron Strucker] added {{graylog-a9c798f5-10e1-4c3b-a6a1-3dfa486b1ea4}{GYaP5Z11RS6HO2tV-FpeSg}{172.18.0.4}{172.18.0.4:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-a9c798f5-10e1-4c3b-a6a1-3dfa486b1ea4}{GYaP5Z11RS6HO2tV-FpeSg}{172.18.0.4}{172.18.0.4:9350}{client=true, data=false, master=false}])
    [2017-04-10 14:40:30,908][WARN ][cluster.routing.allocation.decider] [Baron Strucker] high disk watermark [90%] exceeded on [h2CcecesRl6AKhCg0ROcmw][Baron Strucker][/usr/share/elasticsearch/data/graylog/nodes/0] free: 287.2mb[2.6%], shards will be relocated away from this node
    [2017-04-10 14:41:01,119][WARN ][cluster.routing.allocation.decider] [Baron Strucker] high disk watermark [90%] exceeded on [h2CcecesRl6AKhCg0ROcmw][Baron Strucker][/usr/share/elasticsearch/data/graylog/nodes/0] free: 287.1mb[2.6%], shards will be relocated away from this node
    [2017-04-10 14:41:01,119][INFO ][cluster.routing.allocation.decider] [Baron Strucker] rerouting shards: [high disk watermark exceeded on one or more nodes]
    [2017-04-10 14:41:31,339][WARN ][cluster.routing.allocation.decider] [Baron Strucker] high disk watermark [90%] exceeded on [h2CcecesRl6AKhCg0ROcmw][Baron Strucker][/usr/share/elasticsearch/data/graylog/nodes/0] free: 286.7mb[2.6%], shards will be relocated away from this node
    [2017-04-10 14:42:01,784][WARN ][cluster.routing.allocation.decider] [Baron Strucker] high disk watermark [90%] exceeded on [h2CcecesRl6AKhCg0ROcmw][Baron Strucker][/usr/share/elasticsearch/data/graylog/nodes/0] free: 286.3mb[2.6%], shards will be relocated away from this node
    [2017-04-10 14:42:01,785][INFO ][cluster.routing.allocation.decider] [Baron Strucker] rerouting shards: [high disk watermark exceeded on one or more nodes]
    [2017-04-10 14:42:32,134][WARN ][cluster.routing.allocation.decider] [Baron Strucker] high disk watermark [90%] exceeded on [h2CcecesRl6AKhCg0ROcmw][Baron Strucker][/usr/share/elasticsearch/data/graylog/nodes/0] free: 285.9mb[2.6%], shards will be relocated away from this node
    [2017-04-10 14:43:02,397][WARN ][cluster.routing.allocation.decider] [Baron Strucker] high disk watermark [90%] exceeded on [h2CcecesRl6AKhCg0ROcmw][Baron Strucker][/usr/share/elasticsearch/data/graylog/nodes/0] free: 285.6mb[2.6%], shards will be relocated away from this node
    [2017-04-10 14:43:02,399][INFO ][cluster.routing.allocation.decider] [Baron Strucker] rerouting shards: [high disk watermark exceeded on one or more nodes]
    [2017-04-10 14:43:32,677][WARN ][cluster.routing.allocation.decider] [Baron Strucker] high disk watermark [90%] exceeded on [h2CcecesRl6AKhCg0ROcmw][Baron Strucker][/usr/share/elasticsearch/data/graylog/nodes/0] free: 285.3mb[2.6%], shards will be relocated away from this node
    [2017-04-10 14:44:02,966][WARN ][cluster.routing.allocation.decider] [Baron Strucker] high disk watermark [90%] exceeded on [h2CcecesRl6AKhCg0ROcmw][Baron Strucker][/usr/share/elasticsearch/data/graylog/nodes/0] free: 285mb[2.6%], shards will be relocated away from this node
    [2017-04-10 14:44:02,967][INFO ][cluster.routing.allocation.decider] [Baron Strucker] rerouting shards: [high disk watermark exceeded on one or more nodes]
    [2017-04-10 14:45:02,393][INFO ][cluster.service          ] [Baron Strucker] removed {{graylog-a9c798f5-10e1-4c3b-a6a1-3dfa486b1ea4}{GYaP5Z11RS6HO2tV-FpeSg}{172.18.0.4}{172.18.0.4:9350}{client=true, data=false, master=false},}, reason: zen-disco-node-failed({graylog-a9c798f5-10e1-4c3b-a6a1-3dfa486b1ea4}{GYaP5Z11RS6HO2tV-FpeSg}{172.18.0.4}{172.18.0.4:9350}{client=true, data=false, master=false}), reason(transport disconnected)
    [2017-04-10 14:45:03,357][INFO ][cluster.routing.allocation.decider] [Baron Strucker] rerouting shards: [one or more nodes has gone under the high or low watermark]
    [2017-04-10 14:45:58,434][INFO ][cluster.service          ] [Baron Strucker] added {{graylog-a9c798f5-10e1-4c3b-a6a1-3dfa486b1ea4}{FHF9sEzeR0moZoSs6s9hMA}{172.18.0.4}{172.18.0.4:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-a9c798f5-10e1-4c3b-a6a1-3dfa486b1ea4}{FHF9sEzeR0moZoSs6s9hMA}{172.18.0.4}{172.18.0.4:9350}{client=true, data=false, master=false}])
4

I am a new user, i cannot post more than 32000 characters…
it is not enough to post graloy logs

5

You have over 90% of your disk used.

If you add more displace or delete data Elasticsearch will come up and the messages will be saved and not only stored in the journal

6

Hi Jan,
It was that. But i have increase the disk capacity via LVM…

There is still the same result.

sudo df -h
> Filesystem Size Used Avail Use% Mounted on

    udev                           3,9G     0  3,9G   0% /dev
    tmpfs                          798M   57M  741M   8% /run
    /dev/mapper/vg-root   31G   11G   19G  35% /
    tmpfs                          3,9G  400K  3,9G   1% /dev/shm
    tmpfs                          5,0M     0  5,0M   0% /run/lock
    tmpfs                          3,9G     0  3,9G   0% /sys/fs/cgroup
    /dev/sda2                      473M  119M  330M  27% /boot
    /dev/sda1                      511M  3,6M  508M   1% /boot/efi
    tmpfs                          798M     0  798M   0% /run/user/1000
    none                            31G   11G   19G  35% /var/lib/docker/aufs/mnt/27                                                                                                                                                             3bb8bb67ed9f0d702b52d99e31f0e69be6f751b9182a0298148902fd8517c2
    shm                             64M     0   64M   0% /var/lib/docker/containers/                                                                                                                                                             9eb2136ee69cc03596a4a960c40ba80db7c93ee4a5338e5a78b1f7cdebc03b42/shm
    none                            31G   11G   19G  35% /var/lib/docker/aufs/mnt/8a                                                                                                                                                             726541397bc94414d2d1f0850bb58b30e99ede854c4e30ca6acd3b372fd3a4
    shm                             64M     0   64M   0% /var/lib/docker/containers/                                                                                                                                                             2e039134028c0c6741975f38c4db4a459b9b4c45ff806a47902d4955d1aae91e/shm
    none                            31G   11G   19G  35% /var/lib/docker/aufs/mnt/12                                                                                                                                                             65b660eaca1b5fd11b1df0bb67e2d1e8e09908fbc0b669639a69551c0dd9b4
    shm                             64M     0   64M   0% /var/lib/docker/containers/                                                                                                                                                             e45ef14892a9fb698f9efaceeaa2dc7d0006aef33b7e70225bf1cd6019a71758/shm
7

You can use https://0bin.net or https://gist.github.com to share larger text files.

8

https://0bin.net/paste/iWBfHCs2swMsTmSg#otIB9kK3BMVQZk68rzfzzphke6yd6ApBfCeOxHsPLoj

9

I have deleted my elasticsearch folder an journal foleder.
Killed my dockers and restart them.

It is ok now.