# Saving only part of a log

**URL:** <https://community.graylog.org/t/saving-only-part-of-a-log/25188>\
**Category:** Graylog Central (peer support)\
**Created:** [August 8, 2022, 1:48pm UTC](https://community.graylog.org/t/saving-only-part-of-a-log/25188 "2022-08-08T13:48:32Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alper](https://avatars.discourse-cdn.com/v4/letter/a/a9adbd/32.png) [@Alper](https://community.graylog.org/u/Alper)\
**Post date:** [August 8, 2022, 1:48pm UTC](https://community.graylog.org/t/saving-only-part-of-a-log/25188/1 "2022-08-08T13:48:32Z")

</div>

Graylog 4.3.3

Hi, I’m receiving large logs from a SOPHOS firewall of which I’m only interested in %20 of their content, so saving the whole log is taking so much storage, and as I’m already using extractors to give me the fields I’m interested in, Is there a way to strip the logs and only save specific fields to storage?

---

<div class="post-metadata">

**Author:** ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)\
**Post date:** [August 8, 2022, 2:10pm UTC](https://community.graylog.org/t/saving-only-part-of-a-log/25188/2 "2022-08-08T14:10:56Z")

</div>

You can use the `remove_field()` function in a pipeline to drop the fields you don’t want. I have also used the `regex_replace()` function to remove standardized wording out of some of the larger $message.message results - I want the meat of the message but not the disclaimer that comes with with each one…

That should get you started - if you post more specific information (Obfuscated of course) I might be able to give you a more specific answer! 😃

---

<div class="post-metadata">

**Author:** ![Alper](https://avatars.discourse-cdn.com/v4/letter/a/a9adbd/32.png) [@Alper](https://community.graylog.org/u/Alper)\
**Post date:** [August 9, 2022, 5:11am UTC](https://community.graylog.org/t/saving-only-part-of-a-log/25188/3 "2022-08-09T05:11:10Z")

</div>

Thank you very much, that’s an eye opener, the logs I’m getting look like this:

\<30\>device=“SFW” date=2022-08-09 time=07:58:34 timezone=“+03” device\_name=“XG450” device\_id=C43101J88JXXXXX log\_id=010101600001 log\_type=“Firewall” log\_component=“Firewall Rule” log\_subtype=“Allowed” status=“Allow” priority=Information duration=0 fw\_rule\_id=59 nat\_rule\_id=43 policy\_type=1 user\_name=“” user\_gp=“” iap=18 ips\_policy\_id=0 appfilter\_policy\_id=11 application=“” application\_risk=0 application\_technology=“” application\_category=“” vlan\_id=“” ether\_type=Unknown (0x0000) bridge\_name=“” bridge\_display\_name=“” in\_interface=“XXXXX” in\_display\_interface=“XXXXXX” out\_interface=“Port4” out\_display\_interface=“Port4” src\_mac=E4:XX:4C:00:XX:XX dst\_mac=00:XX:CD:5D:XX:XX src\_ip=XXX.XXX.XXX.XXX src\_country\_code=R1 dst\_ip=XXX.XXX.XXX.XXX dst\_country\_code=R1 protocol=“TCP” src\_port=XXXXX dst\_port=XXXXX sent\_pkts=1 recv\_pkts=1 sent\_bytes=52 recv\_bytes=40 tran\_src\_ip=XXX.XXX.XXX.XXX tran\_src\_port=0 tran\_dst\_ip= tran\_dst\_port=0 srczonetype=“LAN” srczone=“LAN” dstzonetype=“WAN” dstzone=“WAN” dir\_disp=“” connevent=“Stop” connid=“91244XXXX” vconnid=“” hb\_health=“No Heartbeat” message=“” appresolvedby=“Signature” app\_is\_cloud=0

I’m only interested in these fields:

log\_component=“Firewall Rule”  
fw\_rule\_id=59  
user\_name=“”  
user\_gp=“”  
application=“”  
in\_display\_interface=“XXX-XXX”  
src\_mac=E4:XX:4C:00:XX:XX  
dst\_mac=00:XX:CD:5D:XX:XX  
src\_ip=XXX.XXX.XXX.XXX  
dst\_ip=XXX.XXX.XXX.XXX  
src\_port=XXXXX  
dst\_port=XXXXX  
srczone=“LAN”  
dstzone=“WAN”

Since I’m getting millions of these every day, any cuts will benefit greatly, also I have extractors for these values if this is any help…

---

<div class="post-metadata">

**Author:** ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)\
**Post date:** [August 9, 2022, 4:28pm UTC](https://community.graylog.org/t/saving-only-part-of-a-log/25188/4 "2022-08-09T16:28:07Z")

</div>

> [@Alper](#):
>
> also I have extractors for these values

If you can work in extractors to only get the values you want out of the message, that would work - perhaps an well built GROK statement…either in extractors OR in the pipeline

You could leave the extractors in place and then drop the extra fields in the pipeline with the aforementioned functions… or you could drop the extractors all together and separate out your fields in the pipeline where you would only create the fields you want… Someone created a pipeline rule to do just that [here](https://community.graylog.org/t/pipeline-rule-to-pull-specific-key-value-pairs-from-uri/19333) It’s not necessarily efficient… but it works…

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [August 9, 2022, 10:34pm UTC](https://community.graylog.org/t/saving-only-part-of-a-log/25188/5 "2022-08-09T22:34:33Z")

</div>

Hello @Alper

Adding on to @tmacgbay suggestion.

To be honest a pipeline /w regex probably be you best way to create all those fields, I use a lot REGEX extractors for those fields and noticed the resource consumption is greater then just using one pipeline.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [August 23, 2022, 10:35pm UTC](https://community.graylog.org/t/saving-only-part-of-a-log/25188/6 "2022-08-23T22:35:29Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
