1. Describe your incident:
I am trying to get auditd (locally installed) logs to graylog. Installed graylog docs - red_hat_installation (sry, only 2 links). First I did try with rsyslog but it seems to be incompatible with the format (the default logs work but using auditd -only- not). So I went to use filebeat. Is that btw. my best/only option?
2. Describe your environment:
-
OS Information: AlmaLinux 9.2 (5.14.0-284.18.1) clean installed
-
Package Version:
** Server: graylog-server-5.1.3-1.x86_64
** Server: opensearch-2.8.0-1.x86_64
** Server: mongodb-org-6.0.8-1.el9.x86_64 (and its other packages)
** Client: filebeat-7.12.1-1.x86_64 (oss) -
Service logs, configurations, and environment variables:
!!! CLIENT-START !!! filebeat.yml
- type: filestream
enabled: true
id: filebeat-audit-id
paths:
- /var/log/audit/*.log
###
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["10.0.0.20:9200"]
### filebeat test config
Config OK
### filebeat test output
elasticsearch: http://10.0.0.20:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 10.0.0.20
dial up... OK
TLS... WARN secure connection disabled
talk to server... OK
version: 2.8.0
systemctl status filebeat.service
Jul 17 10:25:08 ittest.arxanima.com systemd[1]: filebeat.service: Start request repeated too quickly.
Jul 17 10:25:08 ittest.arxanima.com systemd[1]: filebeat.service: Failed with result 'exit-code'.
Jul 17 10:25:08 ittest.arxanima.com systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..
filebeat -e
2023-07-17T11:48:02.265+0200 INFO instance/beat.go:304 Setup Beat: filebeat; Version: 7.12.1
2023-07-17T11:48:02.265+0200 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.12.1' as ILM is enabled.
2023-07-17T11:48:02.265+0200 INFO eslegclient/connection.go:99 elasticsearch url: http://10.0.0.20:9200
2023-07-17T11:48:02.265+0200 INFO [publisher] pipeline/module.go:113 Beat name: filebeataudit
2023-07-17T11:48:02.266+0200 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2023-07-17T11:48:02.266+0200 INFO instance/beat.go:468 filebeat start running.
2023-07-17T11:48:02.266+0200 INFO memlog/store.go:119 Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=0
runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
netstat -tulpen on client does not show port in use
!!! SERVER-START !!!: Graylog frontend
bind_address: 10.0.0.20
port: 9200
### ^ above Beats input > below "System messages"
Input [Beats/Filebeat/.....] is in state FAILED [bind(..) failed: Address already in use.]
netstat -tulpen
tcp6 0 0 10.0.0.20:9000 :::* LISTEN 981 82104 1974/java
tcp6 0 0 10.0.0.20:9200 :::* LISTEN 982 654709 108490/java
tcp6 0 0 10.0.0.20:9300 :::* LISTEN 982 619269 108490/java
sudo lsof -i :9200
hostnameCOMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 1974 graylog 281u IPv6 703667 0t0 TCP ourhostname:58620->ourhostname:wap-wsp (ESTABLISHED)
java 1974 graylog 285u IPv6 703668 0t0 TCP ourhostname:58634->ourhostname:wap-wsp (ESTABLISHED)
java 1974 graylog 286u IPv6 703669 0t0 TCP ourhostname:58640->ourhostname:wap-wsp (ESTABLISHED)
java 1974 graylog 287u IPv6 703670 0t0 TCP ourhostname:58656->ourhostname:wap-wsp (ESTABLISHED)
java 1974 graylog 293u IPv6 703672 0t0 TCP ourhostname:58668->ourhostname:wap-wsp (ESTABLISHED)
java 108490 opensearch 614u IPv6 654709 0t0 TCP ourhostname:wap-wsp (LISTEN)
java 108490 opensearch 615u IPv6 700544 0t0 TCP ourhostname:wap-wsp->ourhostname:58634 (ESTABLISHED)
java 108490 opensearch 617u IPv6 700545 0t0 TCP ourhostname:wap-wsp->ourhostname:58640 (ESTABLISHED)
java 108490 opensearch 630u IPv6 700546 0t0 TCP ourhostname:wap-wsp->ourhostname:58656 (ESTABLISHED)
java 108490 opensearch 649u IPv6 700547 0t0 TCP ourhostname:wap-wsp->ourhostname:58668 (ESTABLISHED)
java 108490 opensearch 654u IPv6 700543 0t0 TCP ourhostname:wap-wsp->ourhostname:58620 (ESTABLISHED)
/etc/opensearch/opensearch.yml (probably important parts ignoring *names, rest default)
network.host: 10.0.0.20
http.port: 9200
plugins.security.disabled: true
/etc/graylog/server/server.conf
http_bind_address = 10.0.0.20
http_publish_uri = http://10.0.0.20:9000/
elasticsearch_hosts = http://10.0.0.20:9200
mongodb_uri = mongodb://localhost/graylog
3. What steps have you already taken to try and solve the problem?
Switched from original filebeat 8 to filebeat oss as seen here for comptatibilty with opensearch.
Checked configs, logs, restarted services/pcs, checked web on many logs on the way
First did use 127.0.0.1 as openserch bind addr. but then the filebeats test output would fail, this works with local ip as shown above.
4. How can the community help?
I think the problem might be graylog and filebeat to connect on same port (as .f.e. similar here)? But both need to connect to the API so I am not sure how to corrrectly configure it then.
