Configuration of an Extractor on a WAF

[Lexy]

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

Creating extractor for Barracuda [WAF]

Hello everyone,

Im sending this message because i need help, i can’t find a way to create an extractor with a json file for WAF Barracuda last version, i’ve tried to see on the Marketplace, i’ve found an example, but i don’t really understand how does it work, and i’ve tried to import it, but it’s not parsing i’ve also read the documentation about json file, but i don’t understand how to parse logs, when in the logs, as a delimeter it’s a space, and not a caracter like “;” or “,”

Can someone please explain the different steps to create this json file

2. Describe your environment:

• OS Information: WAF / Version 11.0.1.006

• Package Version: Graylog Version / 4.2.6

• Service logs, configurations, and environment variables:

3. What steps have you already taken to try and solve the problem?
i’ve tried to see on the market place, import a json file from it, tried to remplace some parameters and add it to my graylog interface, i’ve read the documentation, saw a lot of videos
4. How can the community help?

It would be very helpful is someone could guide me, i would very appreaciate it, i’m not experimented in parcing logs…

Helpful Posting Tips: Tips for Posting Questions that Get Answers [Hold down CTRL and link on link to open tips documents in a separate tab]

[gsmith]

Hello && Welcome.

Barracuda [WAF] I believe this is a firewall, is this correct?

Correct me if I’m wrong, there are logs sent from Barracuda [WAF] that are in JSON format and you need to create a field/s for these logs?

If you want to use an extractor. If this is correct I might be able to help.

Can you show an example of these logs?

What type of INPUT are you using ( i.e. Syslog UDP/TCP, Raw/Plaintext, GELF, etc…)

If you could explain in greater detail what the end results you want that would be appreciated.

Can you show what has been tried already?

[Lexy]

Hii ! thank you for you respond, i’m glad that you might help me !

So first yes, Barracuda [WAF] is a web firewall ! Second, the logs that are sent from the barracuda are in Syslog NG format, and i would like to create a json extractor to parse those logs correctly ! I saw an option in graylog interface, where i can put json code.

Here is an example of one log just for you to see how they look on graylog without parsing (i’ve put only the beginning, it’s a looongg log)

08:11/69.356 +200 NameFirewall 192.180.65.23 443 91.184.102.100 32694 ‘-‘ ‘-‘ GET TLSv1.3 www.youtube.fr HTTP/1.1 200

As you can see, the delimeter is a space !

For the configuration of the input, i’m using syslog. And i would like to have this kind of results :
Heure : 08:11/69.356 +200

Nom du pare-feu : Pare-feu-secret

Service IP : 192.180.65.23

Port : 443

Adresse Proxy : 91.184.102.100

Proxy port : 32694

Méthode Server : GET

Server protocol : TLSv1.3

Host : www.myefrei.fr

Version serveur : HTTP/1.1

Réponse serveur : 200

I’ve tried to parse with a json file on the market place on graylog, i’ve changed it (the regex rule for example) but it’s not working, and i’m kinda lost…

If you can explain to me how to create the json file, it would mean a lot to me !

Thank you again,

[gsmith]

Hello,
Thanks for the feed back. I am curious and correct me if I’m wrong but the JSON logs are received and indexed, meaning you can see these logs in a search?

Have you tried a different INPUT type/PORT? The reason I stated this is because a GELF UDP input might help create those fields for you. Not sure if this was tested already.

@Lexy

If you can explain to me how to create the json file, it would mean a lot to me !

My apologies , not very good at making a JSON extractor. I try to avoid them as much as possible. I have a tendency to either use a pipeline here or Regex extractor.

@Lexy

I’ve tried to parse with a json file on the market place on graylog, i’ve changed it (the regex rule for example) but it’s not working, and i’m kinda lost…

I can find out what REGEX Extractor are needed from the example of the log you have posted if you want to go that route?

[gsmith]

Ok… so what I did was…

Create a Syslog UDP input.

Then, uploaded this example to my Lab GL server.

@Lexy

08:11/69.356 +200 NameFirewall 192.180.65.23 443 91.184.102.100 32694 ‘-‘ ‘-‘ GET TLSv1.3 www.youtube.fr HTTP/1.1 200

As shown here.

Then I tested a GROK extractor as shown below

Results.

That was a easy way but I prefer REGEX.
Using this command below to grab the IP address

^.+?((?:\d+\.){3}\d+).+$

Test REGEX.

Below you can see both the GROK patterns and the new REGEX fields created.

Hope that helps, sorry its not a JSON extractor but to be honest I seen a pipeline work way better then Extractor since you want multiple fields created.

[Lexy]

Hii !

First of all thank you for every thing that you showed me, it helps me a lot ! I’ve tried your solution, and it actually worked just like you ! So now i’m not going to use a json file, but the grok solution and the regex equation,
But after i’ve tried your solution, i had more questions :
First how can you with the grok solution take the good IP adress ? In the log that i’ve send you, we can see two types of IP adresse, the first one is the Service IP and the second one is the Proxy’s IP,
So how can identify them ?
Another question, how do you take other information like the host (here : www.myefrei.fr)
I’m just trying to understand the logic, and to apply it to the biggest log i have

If you can help me again here, that would be wonderful, and again thank you for everything that you already showed me !

[gsmith]

Hello,

@Lexy

First how can you with the grok solution take the good IP adress ? In the log that i’ve send you, we can see two types of IP adresse, the first one is the Service IP and the second one is the Proxy’s IP,
So how can identify them ?

Couple suggestions on this matter.

You can create your own GROK pattern to separate the specific network addresses.

With multiple IP addresses in a string (message) It might be best to use a pipeline to separate these and create new field/s. HowTo as shown here
and examples shown here

Depending on how the logs from this device are received maybe something like this.

rule "Proxy Server"
when
  has_field("message") AND $message.src_ip == "91.184.102.100"
then
  set_field("proxy_ip", true);
end

It possible to use the Geo-Location Processor found under System/Configuration. probably not the
best for this situation but it will work.

HowTo here

and last you could create a lookup table/s as shown here

Perhaps the JSON might be best, but unfortunately my lack of knowledge on JSON extractors is very limited to NULL

EDIT:
I had time to do a quick test on a pipeline. I just modified one of my rules to get an example.
Steps taken.
1.Create a pipeline. Navigate to System/Pipeline click on "add new pipeline.

2.Create a rule for this pipeline, Click on Manage rules , upper right.

3.Add rule and click apply.

rule "Proxy Server"
when
   contains  (to_string($message.message), "91.184.102.100")
then
  set_field("proxy_ip","91.184.102.100" );
end

4.Go back to Manage Pipelines click on edit. Need to add a couple things in this section which are Connection and Rules. I’m using all messages for my Connection.

5.Click on Add new stage this is where the rule that was just made above is located.

6.Save it , it should look like this.

Results in the red box.

That is about as far as my pipeline knowledge goes. I know there is away to grab specific network addresses. Pipelines can also combine REGEX and/or GROK patterns. @tmacgbay knows more about this then I do.

Hope that helps

EDIT#2:
I’ve been working on some pipeline rules that could be applied. It does need some adjusting thou.

rule "Local address"
When 
cidr_match("192.0.0.0/8", to_ip("192.180.65.23")) &&
 ! cidr_match("191.0.0.0/8", to_ip("192.180.65.23"))
then
set_field("local_address", "192.180.65.23");
end

&&

rule "proxy  address"
When 
cidr_match("91.0.0.0/8", to_ip("91.184.102.100")) &&
! cidr_match("92.0.0.0/8", to_ip("91.184.102.100"))
then
set_field("proxy_address", "91.184.102.100");
end

Results

EDIT#3

@Lexy

Another question, how do you take other information like the host (here : www.myefrei.fr)

This maybe a little long but this might work with a regex extractor

^.+?(?i)\b((?:[a-z][\w-]+:(?:/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}/)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\))+(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s`!()\[\]{};:'".,<>?«»“”‘’])).+$

Results

I found that here

Daring Fireball: An Improved Liberal, Accurate Regex Pattern for Matching URLs

Sorry this is so long. I just realized that last REGEX extractor isnt good, Its picking up incorrect text/strings this works a lot better for an REGEX extractor.

^.+?(www.\s*(\S+)).+?$

[tmacgbay]

I had no idea what it was until just now.

[gsmith]

@tmacgbay Sum it up, @Lexy has logs ingested through Syslog UDP and is wanting specific fields generated. Extractors are Kool for a couple fields but it look like multiple ones are needed. Lexy at first wanted to use JSON but I showed a better way perhaps. Since the logs are coming from a Web Firewall were unable to install a Log shipper. So a pipeline is probable best on that Syslog UDP input.

[gsmith]

@Lexy

I did come up with a unique GROK Pattern. I was experimenting with combining multiple GROK’s which I think would be easy for you.

Pattern:

%{IPV4:local} %{NUMBER:port} %{IP:proxy}

Results:

You can see what I did , but adjustments are needed.
This was only two extractors made.

EDIT: I found something out, since these are default GROK patterns , Just create a GROK extractor on the input used and combine these as shown below. Its a much better outcome. Insure there is a space in-between them.

%{IPV4:local} %{POSINT:port} %{IPV4:proxy}

Results:

Suggestion, I would create a separate INPUT for you WAF device and add these Extractors to that INPUT