Filter subscription in AWS Cloudwatch Logs

Graylog Central (peer support)
1

Description of your problem

I’m streaming logs from AWS Lambda to Graylog (open edition) using Kinesis/Cloudwatch input and it’s works fine if I use the the “Setup Kinesis Automatically” button on Graylog interface. I can see the role, the kinesis stream (with 1 shard) and the subscription in the CloudWatch group.

We know Lambda creates logs with messages like START/END/EXECUTION and I don’t need to stream that because I don’t need those message, I just want my debug messages that I wrote in my code. I tried to create by myself a kinesis/role/subscription using filter to discard these message but I can’t use it on Graylog interface. Graylog says my messages doesn’t have fields like id, message and timestamp, but I can see it my stream.

Someone can help to configure it? Thank you.

Description of steps you’ve taken to attempt to solve the issue

Environmental information

Kubernetes

Operating system information

  • Containers in Kubernetes

Package versions

  • Graylog
  • MongoDB
  • Elasticsearch (deployed on EC2)

NOTE: For all container-based deployments, please include your full, redacted YAML configuration file

NOTE: When posting log output or code snippets (e.g., JSON, YAML, etc.), please surround your code with three backticks like so:

```
 Your code goes here
```

For longer code or configuration bits, please enclose your snippet in a summary block like this:

Summary of your code snippet or config here 
Your code goes inside the triple backticks
2

Adding more information …

I have written some code to look at the kinesis shards and it seems to have the same structure in both cases: Graylog creating kinesis stream or I’m creating it myself.

GRAYLOG KINESIS STREAM:
{“messageType”:“DATA_MESSAGE”,“owner”:“201301409053”,“logGroup”:"/aws/lambda/guru-prd-api-authorization",“logStream”:“2021/10/27/[$LATEST]14413eb225a14f0bbefffb692ecfd95f”,“subscriptionFilters”:[“filter-name”],“logEvents”:[{“id”:“36469168842515009993132838121396095993019798849594654720”,“timestamp”:1635334089415,“message”:“START RequestId: e29db2c1-7a22-485e-89e7-4091da483e90 Version: $LATEST\n”},{“id”:“36469168843630047253059369278472881906652216924893675521”,“timestamp”:1635334089465,“message”:"[GIN] 2021/10/27 - 11:28:09 | 200 | 30.987334ms | | GET “/v2/authorization/authorize”\n"},{“id”:“36469168843630047253059369278472881906652216924893675522”,“timestamp”:1635334089465,“message”:“END RequestId: e29db2c1-7a22-485e-89e7-4091da483e90\n”},{“id”:“36469168843630047253059369278472881906652216924893675523”,“timestamp”:1635334089465,“message”:“REPORT RequestId: e29db2c1-7a22-485e-89e7-4091da483e90\tDuration: 49.07 ms\tBilled Duration: 50 ms\tMemory Size: 128 MB\tMax Memory Used: 41 MB\t\n”},{“id”:“36469168846083129224897737824041810916643536690551521284”,“timestamp”:1635334089575,“message”:“START RequestId: e13db6ec-9285-4c3c-9b0f-364e554abd4c Version: $LATEST\n”},{“id”:“36469168849874255908647943758102883022993758146568192005”,“timestamp”:1635334089745,“message”:"[GIN] 2021/10/27 - 11:28:09 | 200 | 168.194225ms | | GET “/v2/authorization/authorize”\n"},{“id”:“36469168849896556653846474381244418741266406508074172422”,“timestamp”:1635334089746,“message”:“END RequestId: e13db6ec-9285-4c3c-9b0f-364e554abd4c\n”},{“id”:“36469168849896556653846474381244418741266406508074172423”,“timestamp”:1635334089746,“message”:“REPORT RequestId: e13db6ec-9285-4c3c-9b0f-364e554abd4c\tDuration: 169.18 ms\tBilled Duration: 170 ms\tMemory Size: 128 MB\tMax Memory Used: 41 MB\t\n”},{“id”:“36469168850364872303015617467216668824992022099699761160”,“timestamp”:1635334089767,“message”:“START RequestId: 2c150cab-08e5-4d6d-ba93-ebf2ae221b03 Version: $LATEST\n”},{“id”:“36469168850877789442581821799471990345262934414337310729”,“timestamp”:1635334089790,“message”:"[GIN] 2021/10/27 - 11:28:09 | 200 | 16.968155ms | | GET “/v2/authorization/authorize”\n"},{“id”:“36469168850877789442581821799471990345262934414337310730”,“timestamp”:1635334089790,“message”:“END RequestId: 2c150cab-08e5-4d6d-ba93-ebf2ae221b03\n”},{“id”:“36469168850877789442581821799471990345262934414337310731”,“timestamp”:1635334089790,“message”:“REPORT RequestId: 2c150cab-08e5-4d6d-ba93-ebf2ae221b03\tDuration: 17.81 ms\tBilled Duration: 18 ms\tMemory Size: 128 MB\tMax Memory Used: 41 MB\t\n”},{“id”:“36469168851033894658971536161462740373171472944879173644”,“timestamp”:1635334089797,“message”:“START RequestId: daf9760f-8b40-4ef5-83cb-86057b52b2c8 Version: $LATEST\n”},{“id”:“36469168852193533409295128564822597723349187743190155277”,“timestamp”:1635334089849,“message”:"[GIN] 2021/10/27 - 11:28:09 | 200 | 43.321603ms | | GET “/v2/authorization/authorize”\n"},{“id”:“36469168852215834154493659187964133441621836104696135694”,“timestamp”:1635334089850,“message”:“END RequestId: daf9760f-8b40-4ef5-83cb-86057b52b2c8\n”},{“id”:“36469168852215834154493659187964133441621836104696135695”,“timestamp”:1635334089850,“message”:“REPORT RequestId: daf9760f-8b40-4ef5-83cb-86057b52b2c8\tDuration: 50.45 ms\tBilled Duration: 51 ms\tMemory Size: 128 MB\tMax Memory Used: 41 MB\t\n”},{“id”:“36469168854668916126332027733533062451613155870353981456”,“timestamp”:1635334089960,“message”:“START RequestId: b1312a9b-aae2-4596-ae82-e5b3119b3a87 Version: $LATEST\n”},{“id”:“36469168855159532520699701442646848253611419823485550609”,“timestamp”:1635334089982,“message”:"[GIN] 2021/10/27 - 11:28:09 | 200 | 20.521164ms | | GET “/v2/authorization/authorize”\n"}

MY OWN KINESIS STREAM (With filter):
{“messageType”:“DATA_MESSAGE”,“owner”:“201301409053”,“logGroup”:"/aws/lambda/guru-prd-api-authorization",“logStream”:“2021/10/27/[$LATEST]458cb133296241cbb610cf791404df95”,“subscriptionFilters”:[“filter”],"logEvents":[{“id”:“36469165318707358937713483466866547860444135952332685313”,“timestamp”:1635333931402,“message”:"[GIN] 2021/10/27 - 11:25:31 | 200 | 8.787274ms | | GET “/v2/authorization/authorize”\n",“extractedFields”:{“message”:"[GIN] 2021/10/27 - 11:25:31 | 200 | 8.787274ms | | GET “/v2/authorization/authorize”\n"}},{“id”:“36469165323457417665000506196013655852518236953106513925”,“timestamp”:1635333931615,“message”:"[GIN] 2021/10/27 - 11:25:31 | 200 | 53.17141ms | | GET “/v2/authorization/authorize”\n",“extractedFields”:{“message”:"[GIN] 2021/10/27 - 11:25:31 | 200 | 53.17141ms | | GET “/v2/authorization/authorize”\n"}},{“id”:“36469165325709792930052099133308763398055721465210535945”,“timestamp”:1635333931716,“message”:"[GIN] 2021/10/27 - 11:25:31 | 200 | 46.116158ms | | GET “/v2/authorization/authorize”\n",“extractedFields”:{“message”:"[GIN] 2021/10/27 - 11:25:31 | 200 | 46.116158ms | | GET “/v2/authorization/authorize”\n"}},{“id”:“36469165405992475644762342442837349179589822886740033549”,“timestamp”:1635333935316,“message”:"[GIN] 2021/10/27 - 11:25:35 | 200 | 35.332696ms | | GET “/v2/authorization/authorize”\n",“extractedFields”:{“message”:"[GIN] 2021/10/27 - 11:25:35 | 200 | 35.332696ms | | GET “/v2/authorization/authorize”\n"}},{“id”:“36469165442119682866381951932125212781280168526428307473”,“timestamp”:1635333936936,“message”:"[GIN] 2021/10/27 - 11:25:36 | 200 | 19.243233ms | | GET “/v2/authorization/authorize”\n",“extractedFields”:{“message”:"[GIN] 2021/10/27 - 11:25:36 | 200 | 19.243233ms | | GET “/v2/authorization/authorize”\n"}}]}

3

Adding more information…

I see in the Graylog UI this information:
" Additional Information: Unable to map property extractedFields. Known properties include: id, timestamp, message"

And I see in Kinesis stream this field:
{“id”:“36469661889735205797076962308826893080684277354016800770”,“timestamp”:1635356198417,“message”:"[GIN] 2021/10/27 - 17:36:38 | 201 | 157.584729ms | | POST “/v1/registration/document”\n",“extractedFields”:{“message”:"[GIN] 2021/10/27 - 17:36:38 | 201 | 157.584729ms | | POST “/v1/registration/document”\n"}}

There is no way to ignore this field? Is there another way to filter logs on AWS (using subscription) before to send to Graylog?

Thank you.

4

Hello && Welcome

I might be able to help.

Have you tried using a pipeline to drop those messages and filter them into a stream?
I found some posts on these subjects as shown below.

Drop Messages using pipeline

Route to Stream

Hope that helps

1 Like
5

Hei @gsmith, than you so much for your kindness and attention on my problem!

Yeah, pipeline always is a solution, but I’m integrating ~100 lambdas/rds logs and around 1bi requests per month. Even if I configure VPC Endpoint to avoid DataOut to internet, the billing will be an issue because the pipeline will drop message in Graylog and not in the source: Cloudwatch. If I can filter on cloudwatch, I can reduce data transfer by 1/4.

I wrote a lambda to catch the log, remove the “extractFields” and stream to kinesis… It’s working well, but I would like to do this with Kinesis only.

Again, thank you for you reply. I really appreciate it!

1 Like
6

Hello,
Sorry for the delay, Unfortunately I’m not familiar with kinesis.
You could make a simple pipeline rule and attach it to the stream “All Messages”.

Example:

rule “remove cloudwatch”
when
    has_field (“source”) &&
    contains(to_string($message.source), “Cloudwatch”)
then
    drop_message();
end

Then adjust Message Processors Configuration making sure Pipeline is after Message Filter.

  • Message Filter Chain
  • Pipeline Processor
  • GeoIP Resolver
7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.