Help indexing message if possible

Hi all,
I have read the documentation but i’m not excatly sure what im looking for is called. I call it indexing a message.
What I have is this message:

WatchGuard-T35- (2020-01-05T16:15:57) firewall: msg_id=“3000-0149” Allow 2-Unge 0-WAN 40 tcp 20 64 192.168.122.152 aaa.bbb.ccc.ddd 63200 443 offset 5 AF 461166056 win 2048 app_name=“HTTP Protocol over TLS SSL” cat_name=“Network protocols” app_beh_name=“connect” app_id=“94” app_cat_id=“19” app_ctl_disp=“2” msg=“Application identified” (Unge HTTPS-proxy-00)

I would like to sort a search string so i get a list of all different categories after cat_name= within the “” - in this example its Network protocols.

Could someone either help me with the syntax or point me in the right direction?

He @i4admin

you did not wrote where you want to have this kind of indexing …

But with the processing pipelines for example you could make a key-value parserver of this kind of message to have a first separation.

If that was successful you can start splitting more and more our of that.

The following might help you to get an idea.

https://cdn.rawgit.com/jalogisch/OpenSourceDay2018/d3ffdebf/Presentation.html#27

Hi @jan
you are right :slight_smile:
i need to show them in the dashboard. like in this case i want to list all the apps that tricker alerts and how many of them

to explain further.
the rule in question is a firewall alert when application filter is triggered. The alert is based on kategories like “p2p file transfere” is torrent and ftp and such i belive.
I would like to monitor if any of these kategories are denied and allowed i would like to sort it on a per device if possible

ok now i got a little close…

so by clicking on a message from a firewall i got the option to “split and index” which makes a extractor where i can index specific entries in the messege. so i could say i want to index what comes after msg= and it would index what was after that.

This is what i need.

but my problem now is that every piece of information in the log is seperated by space and its not allways the same.

So say i want to index the source IP of the log message. and i sort by space and jump 10 rows ahead to the source ip in the log i get what i want. but the next log message may not have as much information as the former one which mean now the source ip is number 8 when sorting by space.

can i do a more advanced sorting like perhaps look for number dot number dot number dot number? or something like that?

he @i4admin

the processing pipelines Key Value parser is what you want - because you can configure that more than the extractor.

Hi @jan and others

I tried pipelines but thats just way to complex for my needs atm. I’m still new at Graylog.
I really like the extractor because i can do try and see what happens and because they show as filters in the search. I’m sure there is a way to get pipelines in the filters as well but extractors just looks more what i need. Also the documentation states that extractors was specificly made for syslog messenges which is the only messenges i get in my graylog atm.

I tried with the regular expression extractor instead.
I took this expression:
(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})...)(?![0-9])
which i took from the graylog documentation of grok extractors and tried to “try” it on a messge that contained a IP.

I got the error regular expression does not contain any matcher group extract.

I also googled and tried some other ways to express a IPv4 address but i keep getting the same error.
I read the documentation but i’m not seeing what i’m doing wrong. It looks pretty simple to me? maybe im stating my regular expression wrong?

I don’t know it you did your research of Internet carefully, but I found at least 2 sources of extractors for watchguard for graylog:

  1. Grok patterns for extract fields for graylog
    https://gist.github.com/mitchya1/6291aa6173a922d1bf67b2af7f7ceda7

  2. Graylog content pack with dashboard for watchguard
    https://github.com/ThoZed/graylog-cp-watchguard

  3. Fork of previous content pack
    https://github.com/uniQconsulting-ag/graylog.watchguard

Hi @shoothub
I did find the content pack but found it a problem to use since we use docker for our graylog setup. since we have to store a CSV file in /graylog we would need to store the entire graylog installtion on the host which to me made little sense.
correct me if im wrong.
I haven’t found the grok patterns though.

When i read the documentation its specificly says that extractors was made to handle syslog messenges. We intent only to use graylog for syslog. and not only watchguard.
I’m love the saying “give a man fish and his fed for one day, teach a man to fish and he will be fed an entire life”
so i would like to learn how to use the extractors which i’m trying to do instead of grok since extractors are what is supposed to be used for syslog it seams. and in that attempt i apparently am misunderstanding something in relation to how the regex is suppose to be implemented?

Links I posted previously use GROK patterns in extractors.

For example first one:

This URL contains GROK patterns, you can use in your extractors:

  1. Find message you want to extract (like you post in first post):
  2. Expand and click on message arrow on right and select Create extractor for field message - Grok pattern
  3. Use this pattern:

%{NOTSPACE:firewall_name} %{NOTSPACE:UNWANTED} %{NOTSPACE:UNWANTED} msg_id=%{NOTSPACE:msg_id} %{NOTSPACE:action} %{NOTSPACE:inside_interface} %{NOTSPACE:dst_interface} %{BASE10NUM:UNWANTED} %{NOTSPACE:protocol} %{BASE10NUM:UNWANTED} %{BASE10NUM:UNWANTED} %{IPV4:source_ip} %{IPV4:destination_ip} %{BASE10NUM:src_port} %{BASE10NUM:dst_port} %{NOTSPACE:UNWANTED} %{BASE10NUM:UNWANTED} %{NOTSPACE:UNWANTED} %{NOTSPACE:UNWANTED} %{NOTSPACE:UNWANTED} %{NOTSPACE:UNWANTED} app_name=%{QUOTEDSTRING:app_name} cat_name=%{QUOTEDSTRING:category} app_beh_name=%{QUOTEDSTRING:app_beh_name} app_id=%{QUOTEDSTRING:app_id} app_cat_id=%{QUOTEDSTRING:app_cat_id} app_ctl_disp=%{QUOTEDSTRING:app_ctl_disp } msg=%{QUOTEDSTRING:wg_message}

Or little update for you.
Check Named captured only

Click Try against example. This way you show see extracted fields. Name it and save with Create Extractor. Done. Continue with another type of message.

You can also create GROK pattern using simulator in System - Grok pattern - Create pattern and insert Sample data and insert and test field by field. Tris way you use or create desired pattern. After that use it in extractor.

And of course check documentation, to undestand what you are doing:
https://docs.graylog.org/en/3.1/pages/extractors.html#using-grok-patterns-to-extract-data

I think i got it now - thanks for helping :slight_smile:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.