Stream not showing received data

[dansaj]

Hello,
From 19 of july, my stream stopped collecting data.
When I look to Streams menu I can see that stream receive logs, but when click on it I have got " Nothing found in stream 5039-fg100e.sr.lez". When I change time period to 7 days, I can see logs only before 19 of july.
I read that this could be caused by insufficient amount free space on hdd, but I think that I have plenty free space.

df -h

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos_5039–slemonitor00-root 50G 28G 23G 55% /
devtmpfs 3.8G 0 3.8G 0% /dev
tmpfs 3.8G 0 3.8G 0% /dev/shm
tmpfs 3.8G 9.3M 3.8G 1% /run
tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup
/dev/sda2 1014M 222M 793M 22% /boot
/dev/sda1 200M 12M 189M 6% /boot/efi
/dev/mapper/centos_5039–slemonitor00-home 441G 33M 441G 1% /home
tmpfs 769M 0 769M 0% /run/user/0

What can I check now?

Best Regards
Daniel

[jan (Jan Doberstein)]

First check your Graylog and Elasticsearch logfiles for errors.

If that does not give yourself an idea, send a message from the command line and check if you get that message or not.

[dansaj]

In /var/log/graylog-server I got:

2019-06-26T11:24:46.736+02:00 WARN [Messages] Failed to index message: index=<fg100e_0> id=<3d42c070-97f4-11e9-abee-00155d034305> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>
2019-06-26T11:24:46.736+02:00 WARN [Messages] Failed to index message: index=<fg100e_0> id=<3d42c071-97f4-11e9-abee-00155d034305> error=<{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}>

And in /var/log/elasticsearch/gc.log.0 is:

2019-06-17T00:35:44.142+0200: 2896000.120: Total time for which application threads were stopped: 0.0111706 seconds, Stopping threads took: 0.0000701 seconds
2019-06-17T00:35:46.201+0200: 2896002.179: [GC (Allocation Failure) 2019-06-17T00:35:46.201+0200: 2896002.179: [ParNew
Desired survivor size 8716288 bytes, new threshold 1 (max 6)
age 1: 12474480 bytes, 12474480 total
age 2: 228856 bytes, 12703336 total
age 3: 301568 bytes, 13004904 total
age 4: 205720 bytes, 13210624 total
: 143786K->16674K(153344K), 0.0136048 secs] 438561K->311450K(1031552K), 0.0137751 secs] [Times: user=0.02 sys=0.01, real=0.02 secs]
2019-06-17T00:35:46.215+0200: 2896002.192: Total time for which application threads were stopped: 0.0144247 seconds, Stopping threads took: 0.0000780 seconds
2019-06-17T00:35:46.285+0200: 2896002.263: [GC (Allocation Failure) 2019-06-17T00:35:46.285+0200: 2896002.263: [ParNew
Desired survivor size 8716288 bytes, new threshold 6 (max 6)
age 1: 410792 bytes, 410792 total
: 152994K->11041K(153344K), 0.0149755 secs] 447770K->310488K(1031552K), 0.0150928 secs] [Times: user=0.02 sys=0.01, real=0.01 secs]
2019-06-17T00:35:46.301+0200: 2896002.278: Total time for which application threads were stopped: 0.0157179 seconds, Stopping threads took: 0.0000794 seconds
2019-06-17 00:35:47 GC log file has reached the maximum size. Saved as /var/log/elasticsearch/gc.log.0

And in file /var/log/graylog.log is:

[2019-06-26T11:52:52,740][WARN ][o.e.d.c.ParseField ] [6kAMUo2] Deprecated field [auto_generate_phrase_queries] used, replaced by [This setting is ignored, use [type=phrase] instead to make phrase queries out of all text that is within query operators, or use explicitly quoted strings if you need finer-grained control]
[2019-06-26T11:52:52,740][WARN ][o.e.d.c.ParseField ] [6kAMUo2] Deprecated field [split_on_whitespace] used, replaced by [This setting is ignored, the parser always splits on operator]
[2019-06-26T11:52:52,740][WARN ][o.e.d.c.ParseField ] [6kAMUo2] Deprecated field [disable_coord] used, replaced by [disable_coord has been removed]
[2019-06-26T11:52:52,740][WARN ][o.e.d.c.ParseField ] [6kAMUo2] Deprecated field [disable_coord] used, replaced by [disable_coord has been removed]
[2019-06-26T11:53:52,738][WARN ][o.e.d.c.ParseField ] [6kAMUo2] Deprecated field [use_dis_max] used, replaced by [Set [tie_breaker] to 1 instead]
[2019-06-26T11:53:52,738][WARN ][o.e.d.c.ParseField ] [6kAMUo2] Deprecated field [auto_generate_phrase_queries] used, replaced by [This setting is ignored, use [type=phrase] instead to make phrase queries out of all text that is within query operators, or use explicitly quoted strings if you need finer-grained control]
[2019-06-26T11:53:52,738][WARN ][o.e.d.c.ParseField ] [6kAMUo2] Deprecated field [split_on_whitespace] used, replaced by [This setting is ignored, the parser always splits on operator]
[2019-06-26T11:53:52,738][WARN ][o.e.d.c.ParseField ] [6kAMUo2] Deprecated field [use_dis_max] used, replaced by [Set [tie_breaker] to 1 instead]
[2019-06-26T11:53:52,738][WARN ][o.e.d.c.ParseField ] [6kAMUo2] Deprecated field [auto_generate_phrase_queries] used, replaced by [This setting is ignored, use [type=phrase] instead to make phrase queries out of all text that is within query operators, or use explicitly quoted strings if you need finer-grained control]
[2019-06-26T11:53:52,738][WARN ][o.e.d.c.ParseField ] [6kAMUo2] Deprecated field [split_on_whitespace] used, replaced by [This setting is ignored, the parser always splits on operator]
[2019-06-26T11:53:52,738][WARN ][o.e.d.c.ParseField ] [6kAMUo2] Deprecated field [disable_coord] used, replaced by [disable_coord has been removed]
[2019-06-26T11:53:52,738][WARN ][o.e.d.c.ParseField ] [6kAMUo2] Deprecated field [disable_coord] used, replaced by [disable_coord has been removed]
[2019-06-26T11:54:52,742][WARN ][o.e.d.c.ParseField ] [6kAMUo2] Deprecated field [use_dis_max] used, replaced by [Set [tie_breaker] to 1 instead]
[2019-06-26T11:54:52,742][WARN ][o.e.d.c.ParseField ] [6kAMUo2] Deprecated field [auto_generate_phrase_queries] used, replaced by [This setting is ignored, use [type=phrase] instead to make phrase queries out of all text that is within query operators, or use explicitly quoted strings if you need finer-grained control]

I understand only messages from first log, that messagess cannot be indexed because index in read-only mode. But I don’t know why is that…

[jan (Jan Doberstein)]

when your index is read-only Graylog is not able to save new data in Elasticsearch … you should investigate why that had happened and make the index writable again.

More on that topic can be found in the community …