I’m having an issue with java grok not allowing commas in date patterns. I found a post here with my exact issue and it appears the solution is to upgrade to java-grok-0.1.9-graylog to resolve this issue. I found the jar file but I don’t know how to install it or upgrade by just using that jar file. Should I extract the contents of that jar and insert them into the graylog.jar file? It’s just not clear to me how I actually upgrade that particular jar file.
Can someone help me with this?
Thanks!
Jim
Graylog 2.4.6+ceaa7e4
Oracle Corporation 1.8.0_162
Linux 3.10.0-957.12.1.el7.x86_64
Hi @Ponet, it appears I did misunderstand what was happening towards the end of that thread but escaping the comma didn’t work for me either. Here are the stack traces with the comma unescaped and escaped:
Comma Only:
2019-06-20T08:46:29.716-04:00 WARN [MongoDbGrokPatternService] Invalid regular expression syntax for 'MULELOG' with pattern %{WORD:loglevel}%{SPACE}%{MULEDATE:timestamp;date;yyyy-MM-dd HH.mm.ss,SSS}%{SPACE}\[%{DATA:thread}\]%{SPACE}\[event:%{DATA:event}\]%{SPACE}%{DATA:class}\:%{SPACE}%{GREEDYDATA:message}$
java.util.regex.PatternSyntaxException: Illegal repetition near index 13
((null)(null)%{MULEDATE:timestamp;date;yyyy-MM-dd HH.mm.ss,SSS}(null)\[(null)\](null)\[event:(null)\](null)(null)\:(null)(null)$)
^
at java.util.regex.Pattern.error(Pattern.java:1957) ~[?:1.8.0_162]
at java.util.regex.Pattern.closure(Pattern.java:3159) ~[?:1.8.0_162]
at java.util.regex.Pattern.sequence(Pattern.java:2136) ~[?:1.8.0_162]
at java.util.regex.Pattern.expr(Pattern.java:1998) ~[?:1.8.0_162]
at java.util.regex.Pattern.group0(Pattern.java:2907) ~[?:1.8.0_162]
at java.util.regex.Pattern.sequence(Pattern.java:2053) ~[?:1.8.0_162]
at java.util.regex.Pattern.expr(Pattern.java:1998) ~[?:1.8.0_162]
at java.util.regex.Pattern.compile(Pattern.java:1698) ~[?:1.8.0_162]
at java.util.regex.Pattern.<init>(Pattern.java:1351) ~[?:1.8.0_162]
at java.util.regex.Pattern.compile(Pattern.java:1054) ~[?:1.8.0_162]
at com.google.code.regexp.Pattern.buildStandardPattern(Unknown Source) ~[graylog.jar:?]
at com.google.code.regexp.Pattern.<init>(Unknown Source) ~[graylog.jar:?]
at com.google.code.regexp.Pattern.compile(Unknown Source) ~[graylog.jar:?]
at oi.thekraken.grok.api.Grok.compile(Grok.java:391) ~[graylog.jar:?]
at oi.thekraken.grok.api.Grok.compile(Grok.java:328) ~[graylog.jar:?]
at org.graylog2.grok.MongoDbGrokPatternService.validate(MongoDbGrokPatternService.java:106) [graylog.jar:?]
at org.graylog2.grok.MongoDbGrokPatternService.save(MongoDbGrokPatternService.java:78) [graylog.jar:?]
at org.graylog2.rest.resources.system.GrokResource.updatePattern(GrokResource.java:206) [graylog.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [graylog.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Escaped Comma:
2019-06-20T08:47:11.069-04:00 WARN [MongoDbGrokPatternService] Invalid regular expression syntax for 'MULELOG' with pattern %{WORD:loglevel}%{SPACE}%{MULEDATE:timestamp;date;yyyy-MM-dd HH.mm.ss\,SSS}%{SPACE}\[%{DATA:thread}\]%{SPACE}\[event:%{DATA:event}\]%{SPACE}%{DATA:class}\:%{SPACE}%{GREEDYDATA:message}$
java.util.regex.PatternSyntaxException: Illegal repetition near index 13
((null)(null)%{MULEDATE:timestamp;date;yyyy-MM-dd HH.mm.ss\,SSS}(null)\[(null)\](null)\[event:(null)\](null)(null)\:(null)(null)$)
^
at java.util.regex.Pattern.error(Pattern.java:1957) ~[?:1.8.0_162]
at java.util.regex.Pattern.closure(Pattern.java:3159) ~[?:1.8.0_162]
at java.util.regex.Pattern.sequence(Pattern.java:2136) ~[?:1.8.0_162]
at java.util.regex.Pattern.expr(Pattern.java:1998) ~[?:1.8.0_162]
at java.util.regex.Pattern.group0(Pattern.java:2907) ~[?:1.8.0_162]
at java.util.regex.Pattern.sequence(Pattern.java:2053) ~[?:1.8.0_162]
at java.util.regex.Pattern.expr(Pattern.java:1998) ~[?:1.8.0_162]
at java.util.regex.Pattern.compile(Pattern.java:1698) ~[?:1.8.0_162]
at java.util.regex.Pattern.<init>(Pattern.java:1351) ~[?:1.8.0_162]
at java.util.regex.Pattern.compile(Pattern.java:1054) ~[?:1.8.0_162]
at com.google.code.regexp.Pattern.buildStandardPattern(Unknown Source) ~[graylog.jar:?]
at com.google.code.regexp.Pattern.<init>(Unknown Source) ~[graylog.jar:?]
at com.google.code.regexp.Pattern.compile(Unknown Source) ~[graylog.jar:?]
at oi.thekraken.grok.api.Grok.compile(Grok.java:391) ~[graylog.jar:?]
at oi.thekraken.grok.api.Grok.compile(Grok.java:328) ~[graylog.jar:?]
at org.graylog2.grok.MongoDbGrokPatternService.validate(MongoDbGrokPatternService.java:106) [graylog.jar:?]
at org.graylog2.grok.MongoDbGrokPatternService.save(MongoDbGrokPatternService.java:78) [graylog.jar:?]
at org.graylog2.rest.resources.system.GrokResource.updatePattern(GrokResource.java:206) [graylog.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_162]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_162]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) [graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) [graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) [graylog.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) [graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) [graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) [graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) [graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Please let me know what other information I can provide … thanks!
Hmm, looking at your GROK pattern: %{MULEDATE:timestamp;date;yyyy-MM-dd HH.mm.ss,SSS}
From reading that, it looks like you’re attempting to utilise multiple GROK regex patterns for the MULEDATE field, am I right on that?
I’m not sure if you are able to do that but, I haven’t really used GROK all that much so, I may be wrong.
Also, the last section yyyy-MM-dd HH.mm.ss,SSS looks completely wrong to me… From reading it, that regex would be looking for the string “yyyy-MM-dd HH.mm.ss,SSS” with the . character being any non-whitespace character.
It’s not exactly using multiple regex patterns, no. It’s a java date converter you can use to specify the date and time and convert properly so that the incoming timestamp field is correct on inital ingestion and you don’t have to worry about running additional pipelines or extractors to correct date / time issues. You can see examples of folks using it here and here. It’s similar to how you might try to coerce a string into an integer (%{NUMBER:time:int} (?:%{NUMBER:bytes:int}|-)).
For how the grok parser works I think it’s easier to not think of that as a normal regex. Putting it in quotes or escaping the periods doesn’t resolve this particular issue. I think if we were in a pipeline doing something like this with parse_date:
rule "Extract and convert timestamp"
when
has_field("my_application")
then
let message_field = to_string($message.message);
let parsed_fields = grok(pattern: "^%{MY_MESSAGE}", value: message_field, only_named_captures: true);
set_fields(parsed_fields);
let new_date = parse_date(to_string($message.app_timestamp), "yyyy-MM-dd HH:mm:ss,SSS", "en-US", "America/New_York");
set_field("timestamp", new_date);
end
You would be absolutely correct.
But the grok patterns themselves don’t require that - I’m using that exact same format with multiple time stamp formats but they all end in ss.SSS when using milliseconds. I think this is the first one I’ve dealt with using ss,SSS instead which brought me here.
If there are no good solutions I may just end up running it through an additional pipeline rule to see if parse_date will do what I want but I’m trying to avoid introducing an extra processing step.
I was hoping for something a little easier and faster to resolve this issue I’ll see if I can gerry-rig a pipeline to do this in the meantime but I will surely add this to the reasons I need to get upgraded to 3.0.
I’ll see if I can gerry-rig a pipeline to do this in the meantime but I will surely add this to the reasons I need to get upgraded to 3.0.