# Filter subscription in AWS Cloudwatch Logs

**URL:** <https://community.graylog.org/t/filter-subscription-in-aws-cloudwatch-logs/21603>\
**Category:** Graylog Central (peer support)\
**Tags:** pipeline-rules\
**Created:** [October 27, 2021, 2:09pm UTC](https://community.graylog.org/t/filter-subscription-in-aws-cloudwatch-logs/21603 "2021-10-27T14:09:49Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![leandrodamascena](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/leandrodamascena/32/9946_2.png) [@leandrodamascena](https://community.graylog.org/u/leandrodamascena)\
**Post date:** [October 27, 2021, 2:09pm UTC](https://community.graylog.org/t/filter-subscription-in-aws-cloudwatch-logs/21603/1 "2021-10-27T14:09:49Z")

</div>

## Description of your problem

I’m streaming logs from AWS Lambda to Graylog (open edition) using Kinesis/Cloudwatch input and it’s works fine if I use the the “Setup Kinesis Automatically” button on Graylog interface. I can see the role, the kinesis stream (with 1 shard) and the subscription in the CloudWatch group.

We know Lambda creates logs with messages like START/END/EXECUTION and I don’t need to stream that because I don’t need those message, I just want my debug messages that I wrote in my code. I tried to create by myself a kinesis/role/subscription using filter to discard these message but I can’t use it on Graylog interface. Graylog says my messages doesn’t have fields like id, message and timestamp, but I can see it my stream.

Someone can help to configure it? Thank you.

## Description of steps you’ve taken to attempt to solve the issue

## Environmental information

Kubernetes

### Operating system information

- Containers in Kubernetes

### Package versions

- Graylog
- MongoDB
- Elasticsearch (deployed on EC2)

**NOTE:** For all container-based deployments, please include your full, redacted YAML configuration file

**NOTE:** When posting log output or code snippets (e.g., JSON, YAML, etc.), please surround your code with three backticks like so:

```
```
 Your code goes here
```
```

For longer code or configuration bits, please enclose your snippet in a **summary block** like this:

> **Summary of your code snippet or config here**
>
> ```auto
> Your code goes inside the triple backticks
> 
> ```

---

<div class="post-metadata">

**Author:** ![leandrodamascena](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/leandrodamascena/32/9946_2.png) [@leandrodamascena](https://community.graylog.org/u/leandrodamascena)\
**Post date:** [October 27, 2021, 2:17pm UTC](https://community.graylog.org/t/filter-subscription-in-aws-cloudwatch-logs/21603/2 "2021-10-27T14:17:03Z")

</div>

Adding more information …

I have written some code to look at the kinesis shards and it seems to have the same structure in both cases: Graylog creating kinesis stream or I’m creating it myself.

GRAYLOG KINESIS STREAM:  
**{“messageType”:“DATA\_MESSAGE”,“owner”:“201301409053”,“logGroup”:"/aws/lambda/guru-prd-api-authorization",“logStream”:“2021/10/27/[$LATEST]14413eb225a14f0bbefffb692ecfd95f”,“subscriptionFilters”:[“filter-name”],“logEvents”:**[{“ **id** ”:“36469168842515009993132838121396095993019798849594654720”,“ **timestamp** ”:1635334089415,“ **message** ”:“START RequestId: e29db2c1-7a22-485e-89e7-4091da483e90 Version: $LATEST\n”},{“id”:“36469168843630047253059369278472881906652216924893675521”,“timestamp”:1635334089465,“message”:"[GIN] 2021/10/27 - 11:28:09 | 200 | 30.987334ms | | GET “/v2/authorization/authorize”\n"},{“id”:“36469168843630047253059369278472881906652216924893675522”,“timestamp”:1635334089465,“message”:“END RequestId: e29db2c1-7a22-485e-89e7-4091da483e90\n”},{“id”:“36469168843630047253059369278472881906652216924893675523”,“timestamp”:1635334089465,“message”:“REPORT RequestId: e29db2c1-7a22-485e-89e7-4091da483e90\tDuration: 49.07 ms\tBilled Duration: 50 ms\tMemory Size: 128 MB\tMax Memory Used: 41 MB\t\n”},{“id”:“36469168846083129224897737824041810916643536690551521284”,“timestamp”:1635334089575,“message”:“START RequestId: e13db6ec-9285-4c3c-9b0f-364e554abd4c Version: $LATEST\n”},{“id”:“36469168849874255908647943758102883022993758146568192005”,“timestamp”:1635334089745,“message”:"[GIN] 2021/10/27 - 11:28:09 | 200 | 168.194225ms | | GET “/v2/authorization/authorize”\n"},{“id”:“36469168849896556653846474381244418741266406508074172422”,“timestamp”:1635334089746,“message”:“END RequestId: e13db6ec-9285-4c3c-9b0f-364e554abd4c\n”},{“id”:“36469168849896556653846474381244418741266406508074172423”,“timestamp”:1635334089746,“message”:“REPORT RequestId: e13db6ec-9285-4c3c-9b0f-364e554abd4c\tDuration: 169.18 ms\tBilled Duration: 170 ms\tMemory Size: 128 MB\tMax Memory Used: 41 MB\t\n”},{“id”:“36469168850364872303015617467216668824992022099699761160”,“timestamp”:1635334089767,“message”:“START RequestId: 2c150cab-08e5-4d6d-ba93-ebf2ae221b03 Version: $LATEST\n”},{“id”:“36469168850877789442581821799471990345262934414337310729”,“timestamp”:1635334089790,“message”:"[GIN] 2021/10/27 - 11:28:09 | 200 | 16.968155ms | | GET “/v2/authorization/authorize”\n"},{“id”:“36469168850877789442581821799471990345262934414337310730”,“timestamp”:1635334089790,“message”:“END RequestId: 2c150cab-08e5-4d6d-ba93-ebf2ae221b03\n”},{“id”:“36469168850877789442581821799471990345262934414337310731”,“timestamp”:1635334089790,“message”:“REPORT RequestId: 2c150cab-08e5-4d6d-ba93-ebf2ae221b03\tDuration: 17.81 ms\tBilled Duration: 18 ms\tMemory Size: 128 MB\tMax Memory Used: 41 MB\t\n”},{“id”:“36469168851033894658971536161462740373171472944879173644”,“timestamp”:1635334089797,“message”:“START RequestId: daf9760f-8b40-4ef5-83cb-86057b52b2c8 Version: $LATEST\n”},{“id”:“36469168852193533409295128564822597723349187743190155277”,“timestamp”:1635334089849,“message”:"[GIN] 2021/10/27 - 11:28:09 | 200 | 43.321603ms | | GET “/v2/authorization/authorize”\n"},{“id”:“36469168852215834154493659187964133441621836104696135694”,“timestamp”:1635334089850,“message”:“END RequestId: daf9760f-8b40-4ef5-83cb-86057b52b2c8\n”},{“id”:“36469168852215834154493659187964133441621836104696135695”,“timestamp”:1635334089850,“message”:“REPORT RequestId: daf9760f-8b40-4ef5-83cb-86057b52b2c8\tDuration: 50.45 ms\tBilled Duration: 51 ms\tMemory Size: 128 MB\tMax Memory Used: 41 MB\t\n”},{“id”:“36469168854668916126332027733533062451613155870353981456”,“timestamp”:1635334089960,“message”:“START RequestId: b1312a9b-aae2-4596-ae82-e5b3119b3a87 Version: $LATEST\n”},{“id”:“36469168855159532520699701442646848253611419823485550609”,“timestamp”:1635334089982,“message”:"[GIN] 2021/10/27 - 11:28:09 | 200 | 20.521164ms | | GET “/v2/authorization/authorize”\n"}

MY OWN KINESIS STREAM (With filter):  
**{“messageType”:“DATA\_MESSAGE”,“owner”:“201301409053”,“logGroup”:"/aws/lambda/guru-prd-api-authorization",“logStream”:“2021/10/27/[$LATEST]458cb133296241cbb610cf791404df95”,“subscriptionFilters”:[“filter”],"logEvents**":[{“ **id** ”:“36469165318707358937713483466866547860444135952332685313”,“ **timestamp** ”:1635333931402,“ **message** ”:"[GIN] 2021/10/27 - 11:25:31 | 200 | 8.787274ms | | GET “/v2/authorization/authorize”\n",“extractedFields”:{“message”:"[GIN] 2021/10/27 - 11:25:31 | 200 | 8.787274ms | | GET “/v2/authorization/authorize”\n"}},{“id”:“36469165323457417665000506196013655852518236953106513925”,“timestamp”:1635333931615,“message”:"[GIN] 2021/10/27 - 11:25:31 | 200 | 53.17141ms | | GET “/v2/authorization/authorize”\n",“extractedFields”:{“message”:"[GIN] 2021/10/27 - 11:25:31 | 200 | 53.17141ms | | GET “/v2/authorization/authorize”\n"}},{“id”:“36469165325709792930052099133308763398055721465210535945”,“timestamp”:1635333931716,“message”:"[GIN] 2021/10/27 - 11:25:31 | 200 | 46.116158ms | | GET “/v2/authorization/authorize”\n",“extractedFields”:{“message”:"[GIN] 2021/10/27 - 11:25:31 | 200 | 46.116158ms | | GET “/v2/authorization/authorize”\n"}},{“id”:“36469165405992475644762342442837349179589822886740033549”,“timestamp”:1635333935316,“message”:"[GIN] 2021/10/27 - 11:25:35 | 200 | 35.332696ms | | GET “/v2/authorization/authorize”\n",“extractedFields”:{“message”:"[GIN] 2021/10/27 - 11:25:35 | 200 | 35.332696ms | | GET “/v2/authorization/authorize”\n"}},{“id”:“36469165442119682866381951932125212781280168526428307473”,“timestamp”:1635333936936,“message”:"[GIN] 2021/10/27 - 11:25:36 | 200 | 19.243233ms | | GET “/v2/authorization/authorize”\n",“extractedFields”:{“message”:"[GIN] 2021/10/27 - 11:25:36 | 200 | 19.243233ms | | GET “/v2/authorization/authorize”\n"}}]}

---

<div class="post-metadata">

**Author:** ![leandrodamascena](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/leandrodamascena/32/9946_2.png) [@leandrodamascena](https://community.graylog.org/u/leandrodamascena)\
**Post date:** [October 27, 2021, 5:57pm UTC](https://community.graylog.org/t/filter-subscription-in-aws-cloudwatch-logs/21603/3 "2021-10-27T17:57:18Z")

</div>

Adding more information…

I see in the Graylog UI this information:  
" **Additional Information:** Unable to map property extractedFields. Known properties include: id, timestamp, message"

And I see in Kinesis stream this field:  
{“id”:“36469661889735205797076962308826893080684277354016800770”,“timestamp”:1635356198417,“message”:"[GIN] 2021/10/27 - 17:36:38 | 201 | 157.584729ms | | POST “/v1/registration/document”\n",“ **extractedFields** ”:{“message”:"[GIN] 2021/10/27 - 17:36:38 | 201 | 157.584729ms | | POST “/v1/registration/document”\n"}}

There is no way to ignore this field? Is there another way to filter logs on AWS (using subscription) before to send to Graylog?

Thank you.

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [October 27, 2021, 9:53pm UTC](https://community.graylog.org/t/filter-subscription-in-aws-cloudwatch-logs/21603/4 "2021-10-27T21:53:11Z")

</div>

Hello && Welcome

I might be able to help.

> [@leandrodamascena](#):
>
> We know Lambda creates logs with messages like START/END/EXECUTION and I don’t need to stream that because I don’t need those message,

Have you tried using a pipeline to drop those messages and filter them into a stream?  
I found some posts on these subjects as shown below.

[Drop Messages using pipeline](https://community.graylog.org/t/dropping-messages-using-pipelines/11229)

[Route to Stream](https://community.graylog.org/t/pipeline-doest-not-route-to-stream/16534/4)

Hope that helps

---

<div class="post-metadata">

**Author:** ![leandrodamascena](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/leandrodamascena/32/9946_2.png) [@leandrodamascena](https://community.graylog.org/u/leandrodamascena)\
**Post date:** [October 27, 2021, 10:49pm UTC](https://community.graylog.org/t/filter-subscription-in-aws-cloudwatch-logs/21603/5 "2021-10-27T22:49:59Z")

</div>

Hei @gsmith, than you so much for your kindness and attention on my problem!

Yeah, pipeline always is a solution, but I’m integrating ~100 lambdas/rds logs and around 1bi requests per month. Even if I configure VPC Endpoint to avoid DataOut to internet, the billing will be an issue because the pipeline will drop message in Graylog and not in the source: Cloudwatch. If I can filter on cloudwatch, I can reduce data transfer by 1/4.

I wrote a lambda to catch the log, remove the “extractFields” and stream to kinesis… It’s working well, but I would like to do this with Kinesis only.

Again, thank you for you reply. I really appreciate it!

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [November 6, 2021, 12:48am UTC](https://community.graylog.org/t/filter-subscription-in-aws-cloudwatch-logs/21603/6 "2021-11-06T00:48:41Z")

</div>

Hello,  
Sorry for the delay, Unfortunately I’m not familiar with kinesis.  
You could make a simple pipeline rule and attach it to the stream “All Messages”.

Example:

```auto
rule “remove cloudwatch”
when
    has_field (“source”) &&
    contains(to_string($message.source), “Cloudwatch”)
then
    drop_message();
end

```

Then adjust Message Processors Configuration making sure Pipeline is after Message Filter.

- Message Filter Chain
- Pipeline Processor
- GeoIP Resolver

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [November 20, 2021, 12:49am UTC](https://community.graylog.org/t/filter-subscription-in-aws-cloudwatch-logs/21603/7 "2021-11-20T00:49:39Z")

</div>

Closed on November 20, 2021, 12:49am UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.