Anyway…I have a simple syslog stream coming in from a Maltrail Sensor. I am able to GROK the pattern for the src and dst ip, but how do I tell it to extract the fields that is always after the date and | ?
See below: after this field " 2017-02-19| " is the field that would be great to break out into it’s own extracted field.
Given that there are no error messages in the logs of your Graylog node, you could use an absolute time range and set the end point to some time in the future.
This should show messages with an invalid timestamp or invalid timezone information.
you could use following GROK extractor first: %{CEFHEADER:CEFHeader}\|%{DATA:Facility}\|%{DATA:Subfacility}\|%{DATA:version}\|%{DATA:date}\|%{DATA:value1}\|%{DATA:value2}\|%{GREEDYDATA:value3}
After this add a Copy-Input extractor on field value3 (or whatever you rename it to), make it copy/cut to itself and add a Key=Value converter to the extractor.
The reason why the CEF input does not work in your case is, that we are expecting a CEF message to be sent via syslog but your example message is not in syslog format. For example, the PRI field is missing.
I just reproduced this:
org.graylog.plugins.cef.parser.CEFParser$ParserException: This message was not recognized as CEF and could not be parsed.
at org.graylog.plugins.cef.parser.CEFParser.parse(CEFParser.java:71)
at org.graylog.plugins.cef.parser.CEFParserTest.testCustomParse1(CEFParserTest.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
I’m thinking about shipping a parse_cef() pipeline rule to allow more flexible parsing. Would you be willing to give it a try? Should be much easier than using GROK.
You need to use | because | (pipe) is used as OR-operator when using GROK, so you need to escape it with a backslash
I can give a little hint there as well
Make sure that the GeoIP Resolver is last in the processing chain
When you have set it up (added the IP-database and enabled it on the configuration page) it will automatically start to geotag every field having an IP-address
No problem
The GeoIP Resolver needs to be last because it can only check already existing fields for an IP-address, but if it is first in the processing chain there most certainly won’t be a field containing just an IP-address
I guess you can look at the GROK-Pattern and compare it to the String you gave me. Basically you just need to think about using placeholders in the locations where the data that you want is located. But keep in mind that you should be as precise as possible, so don’t use patterns like DATA or GREEDYDATA if you can use more specific ones like IP, DATE, BASE10NUM or whatever
If you dont need the {2017-02-20 21:13:07} 3 150963 part just change the GROK-pattern of text1 to DATA
Well, I just saw that you did it yourself
But you made one mistake that makes Graylog refuse the GROK-pattern. You didn’t escape the { } brackets. These are symbols used by GROK, so they need to be escaped when they are actually inside the string. Else Graylog thinks these brackets are part of a pattern it should parse. Your pattern would be this ````Received: %{DATA:test}{%{TIMESTAMP_ISO8601:date}} %{DATA:value1}{%{DATA:ETRule}} %{IP:DetSource} %{IP:DetDST}```
Notice the TIMESTAMP_ISO8601 pattern instead of DATA on your date field regarding being specific
And you don’t need to make patterns unwanted if you don’t want them (except they are needed to parse the text-structure correctly) . Just make sure you check Named captures only. Graylog will automatically ignore the rest, see:
