Set a new Field in graylog

Hi Graylog Team,

I am new to graylog.

I have a requirement to create a new field as :- real_ip and real port from a raw log .

I want to use x-forwarded-for value as the real ip field and x-forwarded-port as the real port field. Please refer below log as an example.

Example Log:-

“x-forwarded-for”:“”,“x-forwarded-proto”:“https”,“x-forwarded-port”:“443”,“host”:“”,“x-amzn-trace-id”:“Root=1-623e45i6ii003jjifds”,“sec-ch-ua”:“".Not/A)Brand";v="56", "Google Chrome";v="102", "Chromium";v="102"”,“accept”:“application/json;charset=utf-8,/”,“sec-h-a-mobile”:“?0”,“user-agent”:"Mozilla/4.0 (Windows NT 10.0; Win64; x64) AppleWebKit/527.36 (KHTML, like Gecko) Chrome/

From this log we can see “x-forwarded-for” value is =

I want to get this value of x-forwarded-for as real_ip =

Please help me how can i achieve this using a pipeline rule.

Thanks in advance

Hello && welcome @GrAlog_learner

Hope you don’t mind but I moved this post to Graylog Central (i.e., This is for help) the other category templates-and-rules-exchange is for members sharing Templates that they have created.

As for your question you could use a pipeline here


You can look here for examples.


Perhaps this here

Hope that helps

1 Like

Hi @gsmith ,

Thanks for moving my query to correct place .

I am new to graylog so i am not aware where i can post my queries.

I have made partial pipeline rule but i think i am going wrong somewhere . Request you to help me out here please.

Partial Pipeline rule written by me to extract x-forwarded-for as a real-ip field.

rule “Replace x-forwarded-for as Real IP”
let Val= ‘dont know what to mention here. Not sure how this assignment works’
set_field(“real_ip”, to_ip(val)); //


I understand, so here is an example.
NOTE: I have not tested this out yet but perhaps somethings like this.

Explanation, This pipeline states It has the field Message , then using regex to get the IP address after the word x-forwarded-for and place the address under the new field siteName.
I added a DEBUG to this for checking issues. you should be able to see in in Graylog log file.

Shown here.

tail -f /var/log/graylog/server.log

rule "Extract  firewall fields"
  let ipaddress = regex("(x-forwarded-for\\:\\s*(\\S+))", to_string($message.message));
  set_field("siteName", ipaddress["1"]);
  debug (ipaddress);

I create a new rule route to stream

Rule "Route to stream"

Should look something like this.

Hope that helps

I understand that Pipelines are powerful and generally more recommended, but a quick and easy way to extract values into fields using Graylog is to use Extractors. For guidance you may refer to the first example in the GRAYLOG EXTRACTORS EXPLAINED section of the documentation.

Hi @gsmith , @H2Cyber thanks for the help . I was aware about extractors but if i use that then i will turn everything coming into graylog into that field ( I mean we have many streams configured for different purposes) . So i wanted to achieve this using pipeline rule and route the new field only for a single stream . I am routing Security Specific logs to a particular stream without modifying the original stream.

@gsmith , @tmacgbay i did try using the regex . I have few doubts here . Request you to help me on the same:-

  1. Why are we using double slashes in the regex . What flavour of Regex does graylog pipeline support?
  2. Is the regex that we use in extractor different from the regex that we use in Pipeline rules because we don’t use double slashes in extractor . let me know if my understanding is wrong.

I tried to use the regex that you had shared in the pipeline rule but that is capturing a lot of other traffic apart from IP Address . Please refer the below link for reference:-

I tried to fix it using below regex . It is capturing only required field which is IP Address now . Please look below url:-
regex101: build, test, and debug regex.

But in order to make this work in graylog i would have to use double slashes . I am not sure on the double slash part yet. Please have a check and help me out to understand the correct regex format for graylog .

Thanks again for your help .

The short story on the double escape is that it is required in the pipeline rules because of the way that it parses the regex - pipeline escape followed by a regex escape. Nothing fancier than that.

Hi @tmacgbay , @gsmith thanks for the information . But i am still confused with double slashes

My understanding is wherever i am using a “" in normal regex . i would have to use “\” in those places.
For example if my normal regex is :- x-forwarded-for”:“(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})” then the regex for Graylog pipeline should be :- x-forwarded-for"\:“(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})”
Please let me know if i am getting this correct .

Also can you guys let me know if we have any site where we can test double slashed regex expressions supported by Graylog pipeline.

Attaching a pic since my double slashed regex got converted to single slash after posting the reply . Please refer this pic .

It would be a great help if i can get any website where i can test those double slashed expressions

I am not aware of any website that you can test double escaping at. You can just assume that if you are escaping a character in a pipeline to make it a command or literal, you need to add an extra escape.

When you are posting, if you use the </> forum tool, it will treat your test as code and not consider/execute escape sequences… Your regex looks like this when I use the </> tool to highlight it:

My understanding is wherever i am using a "\" in normal regex . i would have to use "\\" in those places. For example if my normal regex is :- x-forwarded-for":"(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})" then the regex for Graylog pipeline should be :- x-forwarded-for"\\:"(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})"
Please let me know if i am getting this correct.

Also… be careful you are using quotes: " and not quotes the latter will cause errors in actual code.

Hi @tmacgbay , @gsmith ,

Awesome . Thanks for your help . Appreciate your Timely responses .

Looking forward to Future Graylog learnings with SME’s like you .

1 Like

Hi @gsmith ,

I am able to do the regex part but i am stuck at getting the data in the new field which i have set.

Please refer the below rule:-

rule "Extract firewall fields" when has_field("message") then let ipaddress = regex("x-forwarded-for\":\"\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}", to_string($message.message)); set_field("IP", ipaddress ); end
when i use set_field(“IP”, ipaddress[“1”] ) in the rule i get no result. Even the field does not get created.

but when i use set_field(“IP”, ipaddress ); i get the below result please refer the below pic:-

kindly have check and please let me know how can i get the ip address thing extracted in the IP field i have used in the rule.


Here is your regex.

let ipaddress = regex("x-forwarded-for\":\"\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}", to_string($message.message)); 

Here is what @tmacgbay suggested.

let ipaddress = regex("x-forwarded-for"\\:"(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})", to_string($message.message));

See the difference??

Also, using “IP” for a field, you may run into issue later on, I would suggest making a unique field, perhaps something like “ip_add”

1 Like

Hi Graylog team ,

I have done slight modifications in the regex you had shared , since it was giving me error. I am able extract the IP but i am getting an extra “0” added in the prefix .

Here is the sample log for your reference:-

{"hostname":"","method":"GET","remote_host":"X.X.X.X","request_headers":{“x-forwarded-for”:“Y.X.Y.X”,“x-forwarded-proto”:“https”,“x-forwarded-port”:“443”,“host”:“”,“x-amzn-trace-id”:“Root=1-AEIOU8NSBDGDY”,“sec-c-ua”:“".Not/A)Brand";v="65", "Google Chrome";v="100", "Chromium";v="100"”,“accept”:“application/json;charset=utf-8,/”,“sec-h-a-mobile”:“?0”,“user-agent”:"M

Regex i have used:-

rule "Extract  firewall fields"
let ipaddress = regex("x-forwarded-for\"\\:\"(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})", to_string($message.message));
  set_field("ip_add", ipaddress);

Output being received

I tried to use array in set field
Like this:-

let ipaddress = regex("x-forwarded-for\"\\:\"(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})", to_string($message.message));
  set_field("ip_add", ipaddress["1"]);

If i use above regex i do not get the ip_add field displayed in the Stream messages.

Please have a check .

Regex starts matches at zero, so it would be:

set_field("ip_add", ipaddress["0"]);

When it shows {"0":"<thing matched>" that’s the first thing matched

1 Like

Hi @tmacgbay and team,

The idea is to extract IP address from x-forwarded-for and then use that IP address in Threat Intel pipeline rules.

If i get output like {"0":"<ip address matched>" then i wont be able to use it in the Threat Intel related Use Cases.

Please let me know how can i achieve this .

In your scenario with regex, using:

set_field("ip_add", ipaddress);

gives you a result in your ip_add field of:

ip_add: {"0":"<ip matched>"}

so you use match ZERO to just get the IP

set_field("ip_add", ipaddress["0"]);

which should result in:

ip_add: <ip matched>

It is referencing the zero in {"0":"<ip matched>"}


Hi @tmacgbay ,

You are awesome.
It’s working now and i am getting only the desired ip address now.


Need one more help regarding the threat intel part.
can i use the below rule using my above extracted field ( i.e ip_add) .

rule "Threat Intel : ip_add"
 set_fields(threat_intel_lookup_ip(to_string($message.ip_addr), "src_addr"));

I am having this doubt because my regex is converting this to to_string so do i need to convert this into to_ip for my threat intel queries to work?

Thanks in Advance


According to Rules Quick Reference it should be of type string:


(Rules Quick Reference is found to the right of where you edit rules.)

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.