Graylog missing messages, fluctuates between 20k and up to 100k per minute

Graylog Central (peer support)
1

Hello Community!

I have searched Google and other forums, including this one, and haven’t found anything similar happening. I am uncertain the cause of this issue, hopefully you can help me figure this one out. Below is a screen shot of what I am seeing, messages are coming in steady anywhere from 20k messages a minute to 100k messages a minute, depending on the time of day. Fairly regularly, Graylog will drop back down to about 20k messages per minute. It appears to affect all sources.

image
image.png732×249 10.3 KB

I am running Graylog on Debian 9 on Proxmox/KVM.
Hardware provisioned to the VM:
20GB RAM
4 CPU’s
700GB on a Samsung 960 EVO

Any help is appreciated, thank you!

2

Hej @manjaro

what is the configuration of your Elasticsearch Cluster? Hardware, HEAP, refresh_index settings

What JAVA Heap did you configured for Graylog?

Did you write any metrics of that System? Did you checked your Graylog and Elasticsearch Logfiles?

regards
Jan

3

Elasticsearch, mongodb, and graylog are all on the same vm. Heap is 8G, I am not familiar with the refresh_index setting so I am guessing whatever default is set for.

Metrics are not set.

Nothing I could see in the logs that I could deduce could cause the issue; here are some things I found, but I am not sure it is related.

/var/log/elasticsearch/graylog-2017-11-13.log
[2017-11-13T16:37:24,075][WARN ][o.e.b.BootstrapChecks ] [7V2qQUE] max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2017-11-13T16:37:24,075][WARN ][o.e.b.BootstrapChecks ] [7V2qQUE] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

I am suspecting perhaps it is my NIC or router. I am seeing traffic dropping during this time as well, so perhaps it is not a Graylog/Elasticsearch issue after all. I guess I assumed it was the server because I never saw this behavior in my test lab setups before I created a production set.

Thank you for your help Jan!

4

@manjaro

is that HEAP for Graylog or Elasticsearch?

For me it looks like your System is underpowered for the amount of messages per seconds you get. Separate Elasticsearch and Graylog and given them more resources and you will not have any issue.

regards
Jan

5

Both are set to 8G. I am not so certain that the system is underpowered. My server metrics during peak operations are showing max 25% cpu usage and 0.5% IO delay, server load tops out at 5% and averages 2%.

6

Is that minimum heap, maximum heap, or both?

Please post the complete JVM configuration for Graylog and Elasticsearch.

Also make sure to leave enough memory for the operating system, so that the disk cache can do its magic.
FWIW, Graylog rarely needs more than 1 or 2 gigabytes of heap memory.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.