Buffer Full and journal disk

Graylog Central (peer support)
1

Good Morning .
I have a 4gb ram and 2 cpu on my instalation and i recibe 4000msg/s , but in 10 seconds the buffer overload , i try to max the ring_size but is not working ,i add another indice too but not lucky. So what do u think i can do , also i dont know where i can assing the size from the elasticsearch , jvm , and SO , i did the default instalation.
Regrats.

2

Are you running Graylog and Elasticsearch on the same machine with 2 CPU cores and 4 GB of memory?

Please post the complete configuration of your Graylog and Elasticsearch nodes and some more details about the hardware specs.

3

Hey dude , in wich file i can edit the size from ElastichSearch and Graylog and SO… Becouse i have a big problem , i recibe like 5000msg/s but the buffer overloads 100% . I have 2 cpu and 4 ram .

4

Yes dude , i use the .ova to install the graylog. I have 20gb disk , here is a screenshot to Graylog nodes.

I dont know if thats what u mean, or when i can find the files to show u … Really aprecciate

5

:arrow_right: http://docs.graylog.org/en/2.4/pages/configuration/file_location.html#omnibus-package

The virtual appliance is not suitable for production use. Additionally, the hardware specs you’ve mentioned (2 CPU cores and 4 GB of RAM) is simply not enough to process and index a throughput of 4000 or 5000 messages per second.

You should think about scaling out your setup:
http://docs.graylog.org/en/2.4/pages/configuration/multinode_setup.html

6

Thanks dude i aprecciate ur help.

So if i want to managment that kind of msg 4000/5000 msg/s , witch hardware specs u recomend me , and where can i set for example jvm = 2gb ram , so 1gb , etc… Because i was reading that could help a lot… psdta: u are not in htb ? :3 i recognized ur turtle.

7

It is difficult to say without knowing the size of the messages, the amount of processing needed and your retention strategy.

If you don’t know the specs you could use virtual machines. With virtual machines you can easily scale up and down.

You could for example start with 2 Graylog servers with 6G RAM and 4 virtual cores, and a load balancer in front of them. If you don’t have high processing power requirements for your logs you could probably easily go with one, but it is good to be able to install updates one node at a time when in production.

For the Elasticsearch you could start with e.g. 2-4 virtual servers with 16G RAM and 4 cores. Be sure to use a proper storage system, so that the ES nodes can have a fast disk access. If you notice then that you don’t have enough power in the ES system, you can easily increase RAM until 64G, vCPUs or add more nodes, depending on what you need.

8

Thanks dude i aprecciate.
Something hilarious happen , i was listed a ACL and i was taken all the logs 3 times , thats why i have 3 millions per minute hahahaha, now i fix that and its normaly :slight_smile: Really apreciate your help , one more question. if i add root: root_timezone = Etc/GMT-3 , and not work. The server is 3 hours more later, i want to change the timesnap from the search …
Thanks

9

Take a look at http://www.joda.org/joda-time/timezones.html and check if you’ve chosen the correct timezone.

10

Yes dude i did , i have the same problem that him.

I see the correct time on the top but wrong over the bot timestamp

image
image.png1384×537 6.5 KB

I edit the server.conf with
root_timezone = Etc/GMT-3

But i dont know how to fix the bot parameter.
Saluds.

11

The “timestamp” field in the message details is always shown in UTC as of Graylog 2.4.x.

This will be fixed in Graylog 3.0.0, see the GitHub issue you’ve linked to.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.