Why are the Pipeline rules not done with an embedded Language?


#1

Hi all,
right now I’m fighting a little with the pipeline rules syntax and have the feeling it is to limited for the field manipulations I need to do.

I can’t influence the log-file format I need to process and some messages need some string processing to create useful fields.
With a ‘complete’ language like jruby I could do this in a heartbeat.
With the rules syntax I’m now forced to do this with java.
This is unfortunately not a good option, because the admins that manage the graylog servers are not java programmers.

I’ve already poked around a little in the graylog code and think this is doable with reasonable effort. I’m also willing to do a POC, but want to hear first, if there is an interest in this feature.

TIA
EDIT TYPO


(Jan Doberstein) #2

first POC was done using LUA for the processing pipeline, but optimizing for speed was an issue. Including the sandboxing. So to have something shippable the decision was made - also with feedback from users - that such a limited language is the better option for 95% of the users.

So feel free to create something new and submit it to the core if you like - or build a plugin that does the modification you like with your own functions.


#3

Thanks for the fast reply

So, what are the sandboxing requirements?
Does Sandboxing mean: Sandboxing to keep variables/code separate from the graylog core?
Or does it mean to prevent ruby code from executing system commands ( with exec)

IMO the first is maybe doable, the second probably not.


(Jan Doberstein) #4

Sandboxing needs to be both. Separate from the Graylog core and prevent to harm the system and creating a potential security hole.


#5

Hi Jan, I played around a little.

The ‘graylog language’ is about 2-3 times as fast as JRuby.
However, if used in mixed mode, it would still be helpful. Do the easy stuff with the ‘graylog language’, the complicated stuff with JRuby.
The Sandboxing could be done with the help of a securitymanager.
Switch to a strict security manager before the exexution of jruby, then switch back to non-restricted after.

Thoughts?


(Jan Doberstein) #6

thank you for your testing. The Core Team will not implement such a feature and it needs to be 100% user-driven.

But from a first perspective, it sounds reasonable.


(system) #7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.