Stream Keeps stopping

Graylog Central (peer support)
,
1

Hello…So I have a stream I think I may be overloading

I keep getting these messages everyone few days

Processing of a stream has been disabled due to excessive processing time. (triggered an hour ago)
The processing of stream 58c9e27b2a0e2e08e95b066c has taken too long for 3 times. To protect the stability of message processing, this stream has been disabled. Please correct the stream rules and reenable the stream

Now…the stream rules are pretty rough…so Is there a better way to accomplish this?

I’m looking to sort out events with certain keywords in them, but drop if other keywords exist. THe rules below work, but seem to be hammering the system

Here are the stream rules

Field EventID must match exactly 4688

Field full_message must match regular expression ((AppData\*\.+?.exe)|RECYCLE.BIN|psexec|powercat|shellcode|powershell.exe.*hidden.*enc|[a-zA-Z0-9/+=]{500}|cscript|wscript)

Field full_message must not match regular expression sdtagged|screenconnect|product_key|EnableWOL|ninite|tsmv1

Ideas?

Thanks

TP

2

I would recommend using pipeline rules instead of the stream rules. I’m not sure how it is on the back end of things, but I have a lot of pipeline rules running over my streams and I have yet to have that issue.

3

Hej @ThomasPowers

similar to what Kris is writing, my personal advice.

The search you are doing in the Stream is complex and heavy to run. Do some processing before you route message into the stream.

One example would be, use a pipeline and at the end you have one field that is true or false and the stream rule check only if that field is set true and you will never have problems with that. Additional you can work better with the data you have.

To get a feeling how this could work, please read this blog posting.