Set a new Field in graylog

GrAlog_learner · July 28, 2022, 7:00pm

Hi Graylog Team,

I am new to graylog.

I have a requirement to create a new field as :- real_ip and real port from a raw log .

I want to use x-forwarded-for value as the real ip field and x-forwarded-port as the real port field. Please refer below log as an example.

Example Log:-

“x-forwarded-for”:“10.2.6.49”,“x-forwarded-proto”:“https”,“x-forwarded-port”:“443”,“host”:“www.example.com”,“x-amzn-trace-id”:“Root=1-623e45i6ii003jjifds”,“sec-ch-ua”:“".Not/A)Brand";v="56", "Google Chrome";v="102", "Chromium";v="102"”,“accept”:“application/json;charset=utf-8,/”,“sec-h-a-mobile”:“?0”,“user-agent”:"Mozilla/4.0 (Windows NT 10.0; Win64; x64) AppleWebKit/527.36 (KHTML, like Gecko) Chrome/102.0.0.0

From this log we can see “x-forwarded-for” value is = 10.2.6.49.

I want to get this value of x-forwarded-for as real_ip = 10.2.6.49

Please help me how can i achieve this using a pipeline rule.

Thanks in advance

gsmith · July 28, 2022, 10:55pm

Hello && welcome @GrAlog_learner

Hope you don’t mind but I moved this post to Graylog Central (i.e., This is for help) the other category templates-and-rules-exchange is for members sharing Templates that they have created.

As for your question you could use a pipeline here

OR

You can look here for examples.

OR

Perhaps this here

Hope that helps

GrAlog_learner · July 29, 2022, 6:21pm

Hi @gsmith ,

Thanks for moving my query to correct place .

I am new to graylog so i am not aware where i can post my queries.

I have made partial pipeline rule but i think i am going wrong somewhere . Request you to help me out here please.

Partial Pipeline rule written by me to extract x-forwarded-for as a real-ip field.

rule “Replace x-forwarded-for as Real IP”
when
(contains(to_string($message.message),x-forwarded-for,true))
then
let Val= ‘dont know what to mention here. Not sure how this assignment works’
set_field(“real_ip”, to_ip(val)); //
route_to_stream(“xxxxxxxxxxxxxxx”);
end

gsmith · July 29, 2022, 10:33pm

Hello,

I understand, so here is an example.
NOTE: I have not tested this out yet but perhaps somethings like this.

Explanation, This pipeline states It has the field Message , then using regex to get the IP address after the word x-forwarded-for and place the address under the new field siteName.
I added a DEBUG to this for checking issues. you should be able to see in in Graylog log file.

Shown here.

tail -f /var/log/graylog/server.log

rule "Extract  firewall fields"
when
  has_field("message") 
then
  let ipaddress = regex("(x-forwarded-for\\:\\s*(\\S+))", to_string($message.message));
  set_field("siteName", ipaddress["1"]);
  debug (ipaddress);
end

I create a new rule route to stream

Rule "Route to stream"
when
    has_field("siteName")
then
    route_to_stream(id:"5d8acba383d72e04cba96317");
end

Should look something like this.

Hope that helps

H2Cyber · July 30, 2022, 12:25pm

I understand that Pipelines are powerful and generally more recommended, but a quick and easy way to extract values into fields using Graylog is to use Extractors. For guidance you may refer to the first example in the GRAYLOG EXTRACTORS EXPLAINED section of the documentation.

GrAlog_learner · July 31, 2022, 7:19pm

gsmith:

rule "Extract  firewall fields"
when
  has_field("message") 
then
  let ipaddress = regex("(x-forwarded-for\\:\\s*(\\S+))", to_string($message.message));
  set_field("siteName", ipaddress["1"]);
  debug (ipaddress);
end

Hi @gsmith , @H2Cyber thanks for the help . I was aware about extractors but if i use that then i will turn everything coming into graylog into that field ( I mean we have many streams configured for different purposes) . So i wanted to achieve this using pipeline rule and route the new field only for a single stream . I am routing Security Specific logs to a particular stream without modifying the original stream.

@gsmith , @tmacgbay i did try using the regex . I have few doubts here . Request you to help me on the same:-

Why are we using double slashes in the regex . What flavour of Regex does graylog pipeline support?
Is the regex that we use in extractor different from the regex that we use in Pipeline rules because we don’t use double slashes in extractor . let me know if my understanding is wrong.

I tried to use the regex that you had shared in the pipeline rule but that is capturing a lot of other traffic apart from IP Address . Please refer the below link for reference:-

I tried to fix it using below regex . It is capturing only required field which is IP Address now . Please look below url:-
regex101: build, test, and debug regex.

But in order to make this work in graylog i would have to use double slashes . I am not sure on the double slash part yet. Please have a check and help me out to understand the correct regex format for graylog .

Thanks again for your help .

tmacgbay · July 31, 2022, 11:14pm

The short story on the double escape is that it is required in the pipeline rules because of the way that it parses the regex - pipeline escape followed by a regex escape. Nothing fancier than that.

GrAlog_learner · August 1, 2022, 6:19am

Hi @tmacgbay , @gsmith thanks for the information . But i am still confused with double slashes

My understanding is wherever i am using a “" in normal regex . i would have to use “\” in those places.
For example if my normal regex is :- x-forwarded-for”:“(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})” then the regex for Graylog pipeline should be :- x-forwarded-for"\:“(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})”
Please let me know if i am getting this correct .

Also can you guys let me know if we have any site where we can test double slashed regex expressions supported by Graylog pipeline.

GrAlog_learner · August 1, 2022, 6:22am

Attaching a pic since my double slashed regex got converted to single slash after posting the reply . Please refer this pic .

It would be a great help if i can get any website where i can test those double slashed expressions

tmacgbay · August 1, 2022, 1:31pm

I am not aware of any website that you can test double escaping at. You can just assume that if you are escaping a character in a pipeline to make it a command or literal, you need to add an extra escape.

When you are posting, if you use the </> forum tool, it will treat your test as code and not consider/execute escape sequences… Your regex looks like this when I use the </> tool to highlight it:

My understanding is wherever i am using a "\" in normal regex . i would have to use "\\" in those places. For example if my normal regex is :- x-forwarded-for":"(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})" then the regex for Graylog pipeline should be :- x-forwarded-for"\\:"(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})"
Please let me know if i am getting this correct.

Also… be careful you are using quotes: " and not quotes ” the latter will cause errors in actual code.

GrAlog_learner · August 1, 2022, 1:47pm

Hi @tmacgbay , @gsmith ,

Awesome . Thanks for your help . Appreciate your Timely responses .

Looking forward to Future Graylog learnings with SME’s like you .

GrAlog_learner · August 11, 2022, 7:56pm

Hi @gsmith ,

I am able to do the regex part but i am stuck at getting the data in the new field which i have set.

Please refer the below rule:-

rule "Extract firewall fields" when has_field("message") then let ipaddress = regex("x-forwarded-for\":\"\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}", to_string($message.message)); set_field("IP", ipaddress ); end
when i use set_field(“IP”, ipaddress[“1”] ) in the rule i get no result. Even the field does not get created.

but when i use set_field(“IP”, ipaddress ); i get the below result please refer the below pic:-

kindly have check and please let me know how can i get the ip address thing extracted in the IP field i have used in the rule.

gsmith · August 11, 2022, 9:30pm

Hello,

Here is your regex.

let ipaddress = regex("x-forwarded-for\":\"\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}", to_string($message.message));

Here is what @tmacgbay suggested.

let ipaddress = regex("x-forwarded-for"\\:"(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})", to_string($message.message));

See the difference??

Also, using “IP” for a field, you may run into issue later on, I would suggest making a unique field, perhaps something like “ip_add”

GrAlog_learner · August 12, 2022, 7:23pm

Hi Graylog team ,

I have done slight modifications in the regex you had shared , since it was giving me error. I am able extract the IP but i am getting an extra “0” added in the prefix .

Here is the sample log for your reference:-

{"hostname":"sample.com","method":"GET","remote_host":"X.X.X.X","request_headers":{“x-forwarded-for”:“Y.X.Y.X”,“x-forwarded-proto”:“https”,“x-forwarded-port”:“443”,“host”:“sample.com”,“x-amzn-trace-id”:“Root=1-AEIOU8NSBDGDY”,“sec-c-ua”:“".Not/A)Brand";v="65", "Google Chrome";v="100", "Chromium";v="100"”,“accept”:“application/json;charset=utf-8,/”,“sec-h-a-mobile”:“?0”,“user-agent”:"M

Regex i have used:-

rule "Extract  firewall fields"
when
  has_field("message") 
then
let ipaddress = regex("x-forwarded-for\"\\:\"(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})", to_string($message.message));
  set_field("ip_add", ipaddress);
end

Output being received

I tried to use array in set field
Like this:-

let ipaddress = regex("x-forwarded-for\"\\:\"(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})", to_string($message.message));
  set_field("ip_add", ipaddress["1"]);

If i use above regex i do not get the ip_add field displayed in the Stream messages.

Please have a check .

tmacgbay · August 12, 2022, 7:45pm

Regex starts matches at zero, so it would be:

set_field("ip_add", ipaddress["0"]);

When it shows {"0":"<thing matched>" that’s the first thing matched

GrAlog_learner · August 12, 2022, 8:28pm

Hi @tmacgbay and team,

The idea is to extract IP address from x-forwarded-for and then use that IP address in Threat Intel pipeline rules.

If i get output like {"0":"<ip address matched>" then i wont be able to use it in the Threat Intel related Use Cases.

Please let me know how can i achieve this .

tmacgbay · August 12, 2022, 9:11pm

In your scenario with regex, using:

set_field("ip_add", ipaddress);

gives you a result in your ip_add field of:

ip_add: {"0":"<ip matched>"}

so you use match ZERO to just get the IP

set_field("ip_add", ipaddress["0"]);

which should result in:

ip_add: <ip matched>

It is referencing the zero in {"0":"<ip matched>"}

GrAlog_learner · August 14, 2022, 5:50pm

Hi @tmacgbay ,

You are awesome.
It’s working now and i am getting only the desired ip address now.

.

Need one more help regarding the threat intel part.
can i use the below rule using my above extracted field ( i.e ip_add) .

rule "Threat Intel : ip_add"
when
 has_field("ip_add")
then
 set_fields(threat_intel_lookup_ip(to_string($message.ip_addr), "src_addr"));
End

I am having this doubt because my regex is converting this to to_string so do i need to convert this into to_ip for my threat intel queries to work?

Thanks in Advance

tmacgbay · August 14, 2022, 6:44pm

Great!!

According to Rules Quick Reference it should be of type string:

(Rules Quick Reference is found to the right of where you edit rules.)

system · August 28, 2022, 6:44pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
New Pipeline rule to copy value and recreate with new name Graylog Central (peer support) pipeline-rules	6	1044	February 20, 2019
Pipeline - Create field with value from another field Graylog Central (peer support) pipeline-rules	3	2257	January 20, 2020
Another try at pipeline rules Graylog Central (peer support) pipeline-rules , debuggingpl	5	1199	August 2, 2017
Problem with to_ip set_field() Graylog Central (peer support) pipeline-rules	7	1221	December 30, 2019
Create a field which name is the value from another on Graylog Central (peer support)	4	1082	October 3, 2018

Set a new Field in graylog

Related Topics