Search for messages with failed grok parser

Graylog Central (peer support)
1

Hello,
sorry for being a noob here, but i tried searching for on how to find messages in a stream where the grok parser did not match. logstash had tags for that, i wonder if i can create a field within grok if nothing matched ?
like | (Add some field) at the end perhaps
or some nicer solution

thanks for any time

2

when your grok matches, you will have new fields.
search for “NOT exists:what_ever_your_fieldname_id” and exclude messages containing that field.

3

Hello, thanks for the reply, but its not that easy, the first part of my grok always matches, i cannot search for nonexistance there,

→ example:
%{CISCO_ASA_TAGGED}
is:
%{POSINT:syslog_pri}>%{CISCOASATIMESTAMP:timestamp2} %{HOSTNAME:source} : ?%?%{CISCO_IOS_TAG:ios_tag}:
does match on
<166>Jun 20 2023 08:23:10 asa-rha : %ASA-6-725016:

the following parts have different fields.

its for a cisco asa: with lots of pattern templates, the many hundret different lines it could throw at graylog as messages can fill completely different fields, maybe sometime even none.

thanks for your time

4

maybe the only solution is to make a new field , implicit, when something matches ? how would i go about that ?
|(%{CISCO_ASA_ACTION:action}( protocol)? %{WORD:protocol} src %{DATA:src_interface}:%{DATA:src_ip}(/%{INT:src_port})?(\(%{DATA:src_fwuser}\))? dst %{DATA:dst_interface}:%{DATA:dst_ip}(/%{INT:dst_port})?(\(%{DATA:dst_fwuser}\))?( \(type %{INT:icmp_type}, code %{INT:icmp_code}\))? by access-group "?%{DATA:policy_id}"? \[%{DATA:hashcode1}, %{DATA:hashcode2}\]) → add something that this matched

|(Group \<%{IPORHOST:group}\> User \<%{USERNAME:user}\> IP \<%{IP:ip}\> IPv4 Address \<%{IP:local_ipv4}\> IPv6 address \<%{IP6:local_ipv6}\> assigned to session) → add something that this matched
|

5

Hello, turns out i had a question mark after the first part (headers) , so it indexed the headers but nothing of the message if it didnt match. e.g.: (( ) | () | () | ())?
So, you are absolutely right i can search for nonexistance of a header field if i omit the question mark.
However out of curiosity my question remains : can i add another field if some grok matched ?

thanks

6

If you use extractors you will not be able to add fields in the moment your grok matched.
If you use pipelines it is possible.

I did a lot of parsing on Cisco ASA. We do it in two steps, as there is a huge number of different messages

This example will cover this message:
%ASA-6-302015: Built {inbound|outbound} UDP connection number for interface_name :real_address /real_port (mapped_address /mapped_port ) [(idfw_user )] to interface_name :real_address /real_port (mapped_address /mapped_port )[(idfw_user )] [(user )]

  1. Step: get the ID of the message-type:
    A Grok Pattern captures only the first part: %ASA-6-302015 → syslog-level:6 and asa_syslog_id:302015
    This happens in stage 1
  2. In Stage two of a pipeline I have a rule like
rule "ASA_302015"
when
  to_string($message.asa_syslog_id) == "302015" 
then
  set_fields(
    grok(
      pattern:"^%{ASA_302015}",
      value:to_string($message.message),
      only_named_captures:true
    )
  );
end

Where I have a Grok Pattern with the name ASA_302015 matching all of the message.
Now I can search for messages not containing any fields from the second stage with the _exists_-Operator.
There are numerous groks and rules like in Step 2, one for each ASA-Pattern. Step 1 only is necessary once.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.