Newbie: Rule not saving


(Roland Hill) #1

A rules newbie here, running graylog 2.4.3

I’ve used graylog before in conjunction with logstash, but have decided this time round to just use graylog and explore grok extractors etc.

Having got those working I realised moving to pipelines and rules was the way to go.

I created this rule (after setting up a stream and pipeline):

    rule "ap1"
    when
      has_field("message")
    then
      let message_field = to_string($message.message);
      let action = grok(pattern: "%{SYSLOGTIMESTAMP:syslog_timestamp} \(""%{SYSLOGHOST:hostname},%{DATA:mac},%{DATA:firmware}""\) %{DATA:program}(?:\[%{POSINT:pid}\])?: %{GREEDYDATA:syslog_message}", value: message_field, only_named_captures: true);;
      set_fields(action);
    end

But it won’t save.

My server logs look like:

2018-03-21T22:04:57.025+13:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.NullPointerException: null
	at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser$RuleAstBuilder.exitNamedArgs(PipelineRuleParser.java:426) ~[?:?]
	at org.graylog.plugins.pipelineprocessor.parser.RuleLangParser$NamedArgsContext.exitRule(RuleLangParser.java:1515) ~[?:?]
	at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:71) ~[?:?]
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:54) ~[?:?]
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
	at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52) ~[?:?]
	at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser.parseRule(PipelineRuleParser.java:170) ~[?:?]
	at org.graylog.plugins.pipelineprocessor.parser.PipelineRuleParser.parseRule(PipelineRuleParser.java:135) ~[?:?]
	at org.graylog.plugins.pipelineprocessor.rest.RuleResource.parse(RuleResource.java:119) ~[?:?]
	at sun.reflect.GeneratedMethodAccessor213.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

I have tried to research this, and am confident it is user error and my lack of understanding, so any hints would be much appreciated.

FYI, this is on my home network. Just trying to learn new things.


Roland


(Jan Doberstein) #2

Hej Roland,

it might be the copy&paste from the forum - but your rule contains a few invalid characters and the editor should show you a reason/error when it is not able to save.


(Roland Hill) #3

Thanks @jan,

I wasn’t getting any errors in the rules window (red crosses), however I created a very simple grok pattern and it does indeed save indicating as you mentioned an issue with my grok line.

I’ll now investigate how I can replicate my working grok extractor into a rule.

Thanks for your input.


(Roland Hill) #4

@jan and community,

Still struggling to get my first rule working (at least it saves now!)

rule "ap1"
when
  has_field("source") && to_string($message.source) == "hillgate"
then
 let message_field = to_string($message.message);
 let action = grok(pattern: "%{SYSLOGTIMESTAMP:syslog_timestamp} %{GREEDYDATA:syslog_message}", value: message_field, only_named_captures: true);;
 set_fields(action);
end

Is there anything noticeably wrong with this rule?

Message look like this:

Thanks again


(Jochen) #5

You can use the debug() function and the pipeline simulator to find out what your rules are doing exactly.


(Jan Doberstein) #6

The GROK pattern does not match your message.

the Pattern "%{SYSLOGTIMESTAMP:syslog_timestamp} %{GREEDYDATA:syslog_message}" can not be found in the field message which starts with hillgate mcad: in this single message and not with a syslog timestamp.

regards
Jan


(Roland Hill) #7

Thanks @jochen, this will allow me to triage my mistakes. I saw this function; not sure why I didn’t use it :thinking:


(Roland Hill) #8

Well how simple was that mistake @jan!

Thank you. The rule is now working and I have a much better understanding of rules.

Thanks for yor patience :+1:


(Roland Hill) #9

SOLVED

Just in case any newbies trip up like I did, below is the working rule I use.

rule "unify usg"
when
  $message.source == "hillgate"
then
 let message_field = to_string($message.full_message);
 let action = grok(pattern: "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:hostname} %{DATA:program} %{GREEDYDATA:syslog_message}", value: message_field, only_named_captures: true);
 set_fields(action);
end

Note, I’m using a UDP Syslog input with “Store full message?(optional)” ticked.


(Jan Doberstein) #10

Hej @nzHillNet

you should mention that you are using a syslog input and have the checkmark for “store full message, for debug only” set.

Did you have more than just the unify gateway or additional hardware? I had created some pipeline rules that deal with the unify messages on a syslog input to set the hostnames and extract all information. Maybe you would like to check them.

regards
Jan


(Roland Hill) #11

Hi @jan,

I did an edit to my solved post as you suggested - thanks.

I have one Unifi USG and one Unifi AP AC-lite. For the AP I followed this:

I havn’t looked to adapt it yet for the USG, just using my previous GROK from logstash.

I would like to try your rules. I’m sure they do a lot more than mine :slight_smile:


Regards,

Roland


(system) #12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.