Graylog is processing messages really slowly

Graylog Central (peer support)
1

Hello everybody,

we are currently trying to use Graylog as a new syslog server and built a POC with it. But at the moment I have got the problem that after a full disk at the weekend it only processes the messages really really slowly. Like 25 Messages per second.
What I did was to delete all indices files after the server was full and creating a new one. And I also deleted the journal files after stopping the graylog services and restarting it.
Elasticsearch seems be fine and in a green state. I already reinstalled everything but I think some configfiles didn’t get deleted because the problem is still there. Do you have any ideas for this problem? I like graylog very much but at the moment it’s not usable.

Best regards,

2
  • Did you checked your logfiles?
  • What did you install and how?
    • what guide did you follow?
  • How does your setup look like?
    • what are the given ressources?
    • what is the ingest rate?
3

Hi,

In the logfiles I don’t see any specific problems.
https://pastebin.com/gdhFrB0T
I followed the guide to install it on a debian server from the offical homepage.
This one:
http://docs.graylog.org/en/2.4/pages/installation/os/debian.html

It all runs one bare metal server with 16 Cores and 64GB of RAM. ES got 24GB and Graylog 12GB.
The ingest rate is at ~4500messages per second.

It worked really good until the whole thing stopped because of the full disk.

Thank you for your help!

Another point is that the server isn’t stressed out. The load average is 0.04 0.09 0.08. So it doesn’t use the hardware at all…

4
  • did you check your elasticsearch logfile?

Did you checked if the (maybe) configured output to the other syslog server able to push the messages over to the other system?

If the target is not reachable that might give you the impression that nothing is processed because every message will run into a timeout …

5

Hey,

the elasticsearch log only shows informational messages and no problems:
https://pastebin.com/Uuzx9CDk
I checked the possible output but there is nothing configured. So I tried to forward the default output but there don’t appear any messages at the other syslog server.
In the default stream I still see the new messages that get processed but really slow.

Here are some pictures maybe its possible to see the problem here…

[Album] Imgur

I am not so deep into linux and graylog but I searched the last days for possible solutions.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.