Search for messages with failed grok parser

sorry for being a noob here, but i tried searching for on how to find messages in a stream where the grok parser did not match. logstash had tags for that, i wonder if i can create a field within grok if nothing matched ?
like | (Add some field) at the end perhaps
or some nicer solution

thanks for any time

when your grok matches, you will have new fields.
search for “NOT exists:what_ever_your_fieldname_id” and exclude messages containing that field.

Hello, thanks for the reply, but its not that easy, the first part of my grok always matches, i cannot search for nonexistance there,

→ example:
%{POSINT:syslog_pri}>%{CISCOASATIMESTAMP:timestamp2} %{HOSTNAME:source} : ?%?%{CISCO_IOS_TAG:ios_tag}:
does match on
<166>Jun 20 2023 08:23:10 asa-rha : %ASA-6-725016:

the following parts have different fields.

its for a cisco asa: with lots of pattern templates, the many hundret different lines it could throw at graylog as messages can fill completely different fields, maybe sometime even none.

thanks for your time

maybe the only solution is to make a new field , implicit, when something matches ? how would i go about that ?
|(%{CISCO_ASA_ACTION:action}( protocol)? %{WORD:protocol} src %{DATA:src_interface}:%{DATA:src_ip}(/%{INT:src_port})?(\(%{DATA:src_fwuser}\))? dst %{DATA:dst_interface}:%{DATA:dst_ip}(/%{INT:dst_port})?(\(%{DATA:dst_fwuser}\))?( \(type %{INT:icmp_type}, code %{INT:icmp_code}\))? by access-group "?%{DATA:policy_id}"? \[%{DATA:hashcode1}, %{DATA:hashcode2}\]) → add something that this matched

|(Group \<%{IPORHOST:group}\> User \<%{USERNAME:user}\> IP \<%{IP:ip}\> IPv4 Address \<%{IP:local_ipv4}\> IPv6 address \<%{IP6:local_ipv6}\> assigned to session) → add something that this matched

Hello, turns out i had a question mark after the first part (headers) , so it indexed the headers but nothing of the message if it didnt match. e.g.: (( ) | () | () | ())?
So, you are absolutely right i can search for nonexistance of a header field if i omit the question mark.
However out of curiosity my question remains : can i add another field if some grok matched ?


If you use extractors you will not be able to add fields in the moment your grok matched.
If you use pipelines it is possible.

I did a lot of parsing on Cisco ASA. We do it in two steps, as there is a huge number of different messages

This example will cover this message:
%ASA-6-302015: Built {inbound|outbound} UDP connection number for interface_name :real_address /real_port (mapped_address /mapped_port ) [(idfw_user )] to interface_name :real_address /real_port (mapped_address /mapped_port )[(idfw_user )] [(user )]

  1. Step: get the ID of the message-type:
    A Grok Pattern captures only the first part: %ASA-6-302015 → syslog-level:6 and asa_syslog_id:302015
    This happens in stage 1
  2. In Stage two of a pipeline I have a rule like
rule "ASA_302015"
  to_string($message.asa_syslog_id) == "302015" 

Where I have a Grok Pattern with the name ASA_302015 matching all of the message.
Now I can search for messages not containing any fields from the second stage with the _exists_-Operator.
There are numerous groks and rules like in Step 2, one for each ASA-Pattern. Step 1 only is necessary once.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.