When evaluating multiple Grok patterns against a message using the extractor or pipeline I get the error below. Each pattern works correctly when its the only pattern.
We were not able to run the grok extraction. Please check your parameters.
Grok Pattern Examples:
Name: COLONACCESSEDRESOURCE
RegEx Pattern: \b(?<=Accessed resources: )[^,]* (All of the Grok are variations of the above pattern)
I am entering the patterns in the extractor like this: %{COLONUSER:UserName} %{COLONCOMPUTERNAME:CallingStationID} %{COLONCONTYPE:ConnectionType} %{COLONACCESSEDRESOURCE:ResourceName} %{COLONACTION:Action}
Same issue with pipeline rules If I enter the rule below nothing will parse, but if I remove all but one pattern it parses.
rule "Parse QNap Logs"
when
has_field("message")
then
let message_field = to_string($message.message);
let QNap = grok(pattern: "%{COLONUSER:UserName} %{COLONCOMPUTERNAME:CallingStationID} %{COLONCONTYPE:ConnectionType} %{COLONACCESSEDRESOURCE:ResourceName} %{COLONACTION:Action}", value: message_field, only_named_captures: true);
set_fields(QNap);
end
I cant find any information indicating what I’m doing wrong please help.
I found an issue on Github here for Graylog 3.x and Grok. In the linked instance it looks like the support for an underscore was dropped… don’t know if that is related. You can double check regex here and grok here online - that may help too…
Graylog is just now at version 5 so you are quite a ways behind… There are quite a few hoops and GUI changes so upgrading won’t be easy… but the longer you let it fester, the harder it gets!
I saw the article about the underscores and I’m not using any so I would really hope that was not the issue. The regex is good and when setting up the GROK patterns in graylog I tested them before saving so I know they evaluate correctly individually. I was thinking there is some separator character that I’m missing.
I have not upgraded because since version 2 every time i do I end up having to rebuild the entire environment because it gets hosed. But at some point I will have to.
So for now I ended up putting each grok pattern in its own rule. Not sure what the difference is in overhead of processing multiple rules instead of one but its working.
Hey, Just throwing my 2 cents in I know this is closed but I was trying to see what difference my Pipe and your pipe had.
“?”
rule "type grok fields"
when
has_field("type") && to_string($message.type) == "Grok fields"
then
// grok the message field
let message_field = to_string($message.message);
let action = grok(pattern: "(%{TIMESTAMP_ISO8601:date-time})?,(%{IPORHOST:client-ip})?,(%{IPORHOST:client-hostname})?,(%{IPORHOST:server-ip})?,(%{IPORHOST:server-hostname})?,(%{GREEDYDATA:source-context})?,(%{GREEDYDATA:connector-id})?,(%{WORD:source})?,(%{WORD:event-id})?,(%{NUMBER:internal-message-id})?,(%{GREEDYDATA:message-id})?,(%{GREEDYDATA:recipient-address})?,(%{GREEDYDATA:recipient-status})?,(%{NUMBER:total-bytes})?,(%{NUMBER:recipient-count})?,(%{GREEDYDATA:related-recipient-address})?,(%{GREEDYDATA:reference})?,(%{GREEDYDATA:message-subject})?,(%{GREEDYDATA:sender-address})?,(%{GREEDYDATA:return-path})?,(%{GREEDYDATA:message-info})?,(%{WORD:directionality})?,(%{GREEDYDATA:tenant-id})?,(%{IPORHOST:original-client-ip})?,(%{IPORHOST:original-server-ip})?,(%{GREEDYDATA:custom-data})?", value: message_field, only_named_captures: true);
set_fields(action);
end
This was something i picked up in the forum while back.
Tried the pipeline rule below based on your suggestion gsmith but sadly I get the same lack of results. Maybe its something with the version, wen I get some time Ill update to Graylog 4 and test it again.
Thanks for the suggestion though.
rule "Parse QNap Logs"
when
has_field("message")
then
let message_field = to_string($message.message);
let QNap = grok(pattern: "(%{COLONUSER:UserName})?, (%{COLONCOMPUTERNAME:CallingStationID})?, (%{COLONCONTYPE:ConnectionType})?, (%{COLONACCESSEDRESOURCE:ResourceName})?, (%{COLONACTION:Action})?,", value: message_field, only_named_captures: true);
set_fields(QNap);
end
