Graylog.conf Processor Settings


(GT) #1

GL 2.2.3

I have a few questions about the following settings in the server.conf file:

processbuffer_processors = 4
outputbuffer_processors = 4

I have 8 cores on my Graylog server, can I only set a max of 8 total across these settings?

My message journal is getting full, I have given my server more ram and increased the cores from before but I haven’t seen a performance increase. I have also updated the JVM settings for ES and GL to be 4gb and 2gb.

What else can I do?

Cheers,

G


(Jake Smith) #2

G,

You can increase the size of the journal on disk in the configuration.

If your journal is getting full are you experiencing spikes of events at a point in time which is overloading the system causing the journal to fill.

Cheers

Jake


(GT) #3

Hi @Magneton,

Our issue is that we can’t process fast enough. We don’t want the journal to become full as then we will begin to lose messages. As you suggested, we could increase the journal size. I would be hesitant to do this as it only buys us breathing space.

I would prefer to tinker with our settings to allow our system to be able to process the messages instead of letting the journal fill up.

I think my next step is to upgrade our GL and ES, were currently on GL 2.2.3 which is about a year old!

Thank you for your advice anyway,

Cheers,

G


(Jake Smith) #4

G,

When I last came across this , I enabled file auditing on a windows share server sending logs into Graylog.

The amount of logs spiked massively (it was a very busy file server) and journal quickly filled up placing the system was under considerable load.

My solution was to increase the journal size to give me more time and then increase ram etc of VM to process messages, none were lost.

I then change the file auditing config to produce less events but the increase in journal size still helped as it provided a buffer to soak up spikes in events.

Cheers

Jake


(Jan Doberstein) #5

did you checked for ES?

In most setups the bottleneck is ES and not Graylog - only when heavy processing is done that might be different.

regards
Jan


(GT) #6

@Magneton I wish we were able to do that. We don’t have that luxury, we need logs to come in and be processed real time as we have alerts that we need triggering as they happen in real life.

@jan I only gave ES 2gb more ram, what settings do you suggest looking at?

Cheers both,

G


#7

Look at the node info in Graylog. If the output buffer is nearly empty, the bottleneck is in processing messages; then you can reduce the number of outputbuffer processors, increase processbuffer processors and optimize your extractors (see for regex backtracking).

If your outputbuffer is full or near full, you need more resources to ES. Consider installing X-Pack to see JVM usage, look at ES logs to see if there is JVM GC warnings, consider making ES a cluster or expanding it.


(GT) #8

Each of the buffers is at about 0%. Once I start the output to send the logs to our other instance of Graylog, the initial Graylog can’t process fast enough.

The output buffer then fills up, I have now changed my settings from:
processbuffer_processors: 20
outputbuffer_processors: 10

to
processbuffer_processors: 10
outputbuffer_processors: 40

Everything works as intended now!

Thanks everyone for the help.

Cheers,

G


(Jochen) #9

Just keep in mind that this Graylog is now using 50 threads to process and output messages.
If you have less than 50 CPU cores, this will induce a performance penalty due to context switching.


(Jake Smith) #10

Jochen,

is there a diagram or document explaining the relationship between performance parameters anywhere?

Cheers

Jake


(Jochen) #11

No, there isn’t any documentation for that yet.


(system) #12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.