Journal growth too much when I enable an extractor

Graylog Central (peer support)
1

Hello,
When I enable an extractor to split mod_security logs, After 1 or 2 hours, my journal grow up too much, and can’t consume all logs.
I already have a lot of extractors from years ago without problem, But this one crash Graylog :frowning:

I added RAM to graylog2 process (2Go of XmX right now), but same problem.

NNothing on logs.I have to restart graylog, and disable extractor to resume.
The extractor is a grok pattern:

[%{HTTPDERROR_DATE:timestamp}] [(%{WORD:module}|):%{LOGLEVEL:loglevel}] [pid %{POSINT:pid}(:tid %{POSINT:tid}|)]( (%{POSINT:proxy_errorcode})%{DATA:proxy_errormessage}:)?( [client %{IPORHOST:client}:%{POSINT:clientport}])?( [client %{IPORHOST:client2}]) %{GREEDYDATA:alerte} [file “%{PATH:rule_file}”] [line “%{POSINT:line}”] [id “%{POSINT:id}”] [msg “%{GREEDYDATA:message}”] [data “%{IP:ip_server}”] [severity “%{LOGLEVEL:loglevel2}”] [ver “%{DATA:version_crs}”]( [tag “%{DATA:tag1}”]|)( [tag “%{DATA:tag2}”]|)( [tag “%{DATA:tag3}”]|)( [tag “%{DATA:tag4}”]|)( [tag “%{DATA:tag5}”]|)( [tag “%{DATA:tag6}”]|)( [tag “%{DATA:tag7}”]|)( [tag “%{DATA:tag8}”]|)( [tag “%{DATA:tag9}”]|)( [tag “%{DATA:tag10}”]|) [hostname “%{IP:ip_server2}”] [uri “%{DATA:uri}”] [unique_id “%{DATA:unique_id}”]

I’ve tryed to send logs direct in json, to have field splited, but I use filebeat (and logstash output) to send logs, and don’t know how to configure it.

2

Here is an earlier post about GROK performance with a solution that seems related:

There are a couple things that you can do to increase performance such as using NOTSPACE instead of DATA (if possible) and strictly defining the beginning (^) and end ($) of your GROK

3

I double check the journal usage:
1 hour after I done your advices, with start, and end regex, and NOTSPACE, the Process buffer start to raise (cpu 100% with graylog process)
Once Process buffer up to 65536 (100%), Journal start to growth :confused:

From 30 min ago, I have only 4700 msg in this input.
I don’t understand why there is a bottleneck :confused:

4

I have also had GROK hang a processor buffer- on the NODE page choose more actions then get processor-buffer dump. Normally I see those as idle but when I was getting system hangs it would show the message it was hung up on…

5

Ok, I will test this thursday. Thanks

6

OK I’ve got problematic message. If I try this with extractor simulate, It bug.
I should rebuild extractor.
I hope this will be ok.

7

This is ok, But I had to split it in 3 extractors :(.

Thanks for tips

8

You could do the work in a pipeline rather than extractor - pipelines would allow you to narrow where the work is done a little better.

1 Like
9

Indeed, with pipeline, we have more granularity.
Thanks

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.