Not able to push ALB logs to Graylog

Graylog Tech Challenges
,
1

Hi,
I am not able to push Access logs(Logs present on the server) for Application Load Balancer in the Graylog.

I am using the collector sidecar for pushing the logs. Below is my collector configuration:

Screen Shot 2021-08-31 at 11.44.32 PM
Screen Shot 2021-08-31 at 11.44.32 PM460×1010 56.3 KB

And the Regex I am using is ^\d{4}-\d{2}-\d{2}

Here is the log format:

h2 2021-08-29T07:29:58.746544Z app/app-test-com/32bd67569317b9a8e3a5 65.114.117.102:61351 10.0.24.35:80 0.000 0.114 0.000 200 200 383 201 "POST https://app.test.com:443/con/embedded/dbserviceV2 HTTP/2.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.45715.159 Safari/537.36" ECDHE-RSA-AES128-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:us-east-1:578066756415:targetgroup/app-test-com-v1-80/92041a284e0bbc0d "Root=1-612b3776-5db6bgty67hgc009c20e4a3a40" "app.test.com" "arn:aws:acm:us-east-1:5756981045415:certificate/4fb3t6y58-5fc1-48ad-9072-16e3at6y7u32" 0 2021-08-29T07:29:58.632000Z "forward" "-" "-" "10.0.24.35:80" "200" "-" "-"

Here is the filebeat configuration:

filebeat:
  prospectors:
  - encoding: plain
    exclude_files: []
    fields:
      collector_node_id: alb-logs
      gl2_source_collector: |-
        bb0959878-3090-458e-923b-60d0da169dca
        #2456785-2fa5-47bd-84b4-bc0e9436c9bf
      type: log
    ignore_older: 0
    multiline:
      match: after
      negate: true
      pattern: ([^ ]*)
    paths:
    - /var/log/alb/alb.log
    scan_frequency: 10s
    symlinks: false
    tail_files: true
    type: log
  - encoding: plain
    exclude_files: []
    fields:
      collector_node_id: alb-logs
      gl2_source_collector: |-
        bb0567008-3090-458e-923b-60d0da169dca
        #2a345285-2fa5-47bd-84b4-bc0e9436c9bf
      type: log
    ignore_older: 0
    multiline:
      match: after
      negate: false
      pattern: ^(?P<h2>[^\s]+)\s*
    paths:
    - /var/log/haproxy/haproxy.log
    scan_frequency: 10s
    symlinks: false
    tail_files: true
    type: log
output:
  logstash:
    hosts:
    - 10.0.xx.xx:5044
path:
  data: /var/cache/graylog/collector-sidecar/filebeat/data
  logs: /var/log/graylog/collector-sidecar
tags:
- alb-logs

Graylog Version: 3.0.2

2

Hello @Tafsir_Alam

I’ve been looking over you post for a couple hours. It has been a long time since I seen/used Collector Sidecar. Since your using Graylog 3.0, by chance have you thought about upgrade to the new sidecar? I have found it to be very easy to setup.
You can find it here

image
image1290×596 55.5 KB

As for your issue, did you check permissions of the files/folders to see if there accessible?
Do you have a firewall or Selinux enabled?

I’m assuming this is a package installment (YUM, APT) and perhaps its on CentOS?
If so, do you see anything in the log files that may pertain to this issue?