Filebeat for auditd logs using opensearch/graylog fails on address binding

1. Describe your incident:
I am trying to get auditd (locally installed) logs to graylog. Installed graylog docs - red_hat_installation (sry, only 2 links). First I did try with rsyslog but it seems to be incompatible with the format (the default logs work but using auditd -only- not). So I went to use filebeat. Is that btw. my best/only option?

2. Describe your environment:

  • OS Information: AlmaLinux 9.2 (5.14.0-284.18.1) clean installed

  • Package Version:
    ** Server: graylog-server-5.1.3-1.x86_64
    ** Server: opensearch-2.8.0-1.x86_64
    ** Server: mongodb-org-6.0.8-1.el9.x86_64 (and its other packages)
    ** Client: filebeat-7.12.1-1.x86_64 (oss)

  • Service logs, configurations, and environment variables:

!!! CLIENT-START !!! filebeat.yml

- type: filestream
  enabled: true
  id: filebeat-audit-id
    - /var/log/audit/*.log
  # Array of hosts to connect to.
  hosts: [""]
### filebeat test config
Config OK
### filebeat test output
  parse url... OK
    parse host... OK
    dns lookup... OK
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 2.8.0

systemctl status filebeat.service

Jul 17 10:25:08 systemd[1]: filebeat.service: Start request repeated too quickly.
Jul 17 10:25:08 systemd[1]: filebeat.service: Failed with result 'exit-code'.
Jul 17 10:25:08 systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..

filebeat -e

2023-07-17T11:48:02.265+0200	INFO	instance/beat.go:304	Setup Beat: filebeat; Version: 7.12.1
2023-07-17T11:48:02.265+0200	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'filebeat-7.12.1' as ILM is enabled.
2023-07-17T11:48:02.265+0200	INFO	eslegclient/connection.go:99	elasticsearch url:
2023-07-17T11:48:02.265+0200	INFO	[publisher]	pipeline/module.go:113	Beat name: filebeataudit
2023-07-17T11:48:02.266+0200	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
2023-07-17T11:48:02.266+0200	INFO	instance/beat.go:468	filebeat start running.
2023-07-17T11:48:02.266+0200	INFO	memlog/store.go:119	Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=0
runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort

netstat -tulpen on client does not show port in use

!!! SERVER-START !!!: Graylog frontend

port: 9200
###  ^ above Beats input > below "System messages"
Input [Beats/Filebeat/.....] is in state FAILED [bind(..) failed: Address already in use.]

netstat -tulpen

tcp6       0      0         :::*                    LISTEN      981        82104      1974/java           
tcp6       0      0         :::*                    LISTEN      982        654709     108490/java         
tcp6       0      0         :::*                    LISTEN      982        619269     108490/java        

sudo lsof -i :9200

java      1974    graylog  281u  IPv6 703667      0t0  TCP ourhostname:58620->ourhostname:wap-wsp (ESTABLISHED)
java      1974    graylog  285u  IPv6 703668      0t0  TCP ourhostname:58634->ourhostname:wap-wsp (ESTABLISHED)
java      1974    graylog  286u  IPv6 703669      0t0  TCP ourhostname:58640->ourhostname:wap-wsp (ESTABLISHED)
java      1974    graylog  287u  IPv6 703670      0t0  TCP ourhostname:58656->ourhostname:wap-wsp (ESTABLISHED)
java      1974    graylog  293u  IPv6 703672      0t0  TCP ourhostname:58668->ourhostname:wap-wsp (ESTABLISHED)
java    108490 opensearch  614u  IPv6 654709      0t0  TCP ourhostname:wap-wsp (LISTEN)
java    108490 opensearch  615u  IPv6 700544      0t0  TCP ourhostname:wap-wsp->ourhostname:58634 (ESTABLISHED)
java    108490 opensearch  617u  IPv6 700545      0t0  TCP ourhostname:wap-wsp->ourhostname:58640 (ESTABLISHED)
java    108490 opensearch  630u  IPv6 700546      0t0  TCP ourhostname:wap-wsp->ourhostname:58656 (ESTABLISHED)
java    108490 opensearch  649u  IPv6 700547      0t0  TCP ourhostname:wap-wsp->ourhostname:58668 (ESTABLISHED)
java    108490 opensearch  654u  IPv6 700543      0t0  TCP ourhostname:wap-wsp->ourhostname:58620 (ESTABLISHED)

/etc/opensearch/opensearch.yml (probably important parts ignoring *names, rest default)
http.port: 9200 true


http_bind_address =
http_publish_uri =
elasticsearch_hosts =
mongodb_uri = mongodb://localhost/graylog

3. What steps have you already taken to try and solve the problem?
Switched from original filebeat 8 to filebeat oss as seen here for comptatibilty with opensearch.
Checked configs, logs, restarted services/pcs, checked web on many logs on the way

First did use as openserch bind addr. but then the filebeats test output would fail, this works with local ip as shown above.

4. How can the community help?
I think the problem might be graylog and filebeat to connect on same port (as .f.e. similar here)? But both need to connect to the API so I am not sure how to corrrectly configure it then.

Any of the beats will need to send data to graylog using the logstash output:

   hosts: ["hostname.domain.tld:5044"]

and will send to a beat input on the graylog server using the corresponding TCP port, which by default is 5044.


Let us know if that doesn’t resolve your issue.

Thank you, might be the issue. Will test shortly but it makes sense.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.