Sidecar configuration for auditbeat and journalbeat [Linux hosts]

[cesq]

Hello! I am having an issue with writing the sidecar configuration for auditbeat and journalbeat.
I want to test out filebeat, auditbeat and journalbeat and for that I need all of these to work.
Now I have filebeat pretty much figured out, as there’s tons of official documentation about it.
However I cannot figure out how to configure sidecars for auditbeat and journalbeat nor do I know how to install and configure them on the host from which the data will be collected.

I am aware of the documentation available from elasticsearch, but I am unable to use it to my advantage, as there’s no mention of graylog in there whatsoever and everything that I’ve found is related to Windows hosts.

I have already configured the Beats Input. My graylog version is 4.3.7

[gsmith]

Hello @cesq

Yeah, I also have not found much on those log shippers as for configuring it on Graylog.

I have install Packetbeat perhaps it can give you an Idea what to look for. I took some documentations from Graylog and from Elasticsearch for the beats log shipper configuration.

Example below I user Graylog Sidecar with Packet Beat.

Created Input Beat port 5066
Installed packet beat from here on remote Linux node
Adjusted Graylog sidecar configuration file to pick up Packetbeat

# A list of binaries which are allowed to be executed by the Sidecar. An empty list disables the whitelist feature.
# Wildcards can be used, for a full pattern description see https://golang.org/pkg/path/filepath/#Match
# Example:
#     collector_binaries_whitelist:
#       - "/usr/bin/filebeat"
#       - "/opt/collectors/*"
#
# Example disable whitelisting:
#     collector_binaries_whitelist: []
#
# Default:
collector_binaries_whitelist:
  - "/usr/bin/filebeat"
  - "/usr/bin/packetbeat"
#  - "/usr/bin/metricbeat"
#  - "/usr/bin/heartbeat"
#  - "/usr/bin/auditbeat"
#  - "/usr/bin/journalbeat"
  - "/usr/share/filebeat/bin/filebeat"
  - "/usr/share/packetbeat/bin/packetbeat"
#  - "/usr/share/metricbeat/bin/metricbeat"
#  - "/usr/share/heartbeat/bin/heartbeat"
#  - "/usr/share/auditbeat/bin/auditbeat"
#  - "/usr/share/journalbeat/bin/journalbeat"
#  - "/usr/bin/nxlog"
#  - "/opt/nxlog/bin/nxlog"

On Graylog GUI, i made the template and attached to the remote node.

Packetbeat configuration

packetbeat.interfaces.device: any
packetbeat.interfaces.internal_networks:
  - private
packetbeat.flows:
  timeout: 30s
  period: 10s
packetbeat.protocols:
- type: icmp
  enabled: true
- type: amqp
  ports: [5672]
- type: cassandra
  ports: [9042]
- type: dhcpv4
  ports: [67, 68]
- type: dns
  ports: [53]
- type: http
  ports: [80, 8080, 8000, 5000, 8002]
- type: memcache
  ports: [11211]
- type: mysql
  ports: [3306,3307]
- type: pgsql
  ports: [5432]
- type: redis
  ports: [6379]
- type: thrift
  ports: [9090]
- type: mongodb
  ports: [27017]
- type: nfs
  ports: [2049]
- type: tls
  ports:
    - 443   # HTTPS
    - 993   # IMAPS
    - 995   # POP3S
    - 5223  # XMPP over SSL
    - 8443
    - 8883  # Secure MQTT
    - 9243  # Elasticsearch
- type: sip
  ports: [5060]
  _source.enabled: true
output.logstash:
  hosts: ["10.10.10.10:5066"]
processors:
  - # Add forwarded to tags when processing data from a network tap or mirror.
    if.contains.tags: forwarded
    then:
      - drop_fields:
          fields: [host]
    else:
      - add_host_metadata: ~
  - add_cloud_metadata: ~
  - detect_mime_type:
      field: http.request.body.content
      target: http.request.mime_type
  - detect_mime_type:
      field: http.response.body.content
      target: http.response.mime_type
logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/packetbeat
  name: packetbeat
  keepfiles: 7
  permissions: 0640

Good after that, came in fine.

[cesq]

@gsmith Thanks, that’ll help me a bit. Still, if anyone that comes here, has some experience with this topic, I’d love to get some info, thanks!

[cesq]

@gsmith So I made a simple configuration for auditbeat, however it keeps failing. I know that you haven’t done this, but maybe you can spot a mistake here?

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
output.logstash:
   hosts: ["x.x.x.x:5044"]
path:                                                               
  data: /var/lib/graylog-sidecar/collectors/auditbeat/data      
  logs: /var/lib/graylog-sidecar/collectors/auditbeat/log                      
tags: 
    - linux                                                     
    - auditbeat                                                     
auditbeat.modules:
- module: file_integrity
  paths:
  - /bin
  - /usr/bin
  - /sbin
  - /usr/sbin
  - /etc
  - /var/log
- module: system
  datasets:
    - host      
    - process  
    - login   
    - process
    - socket
    - user
    
state.period: 12s
user.detect_password_changes: true                                  

[cesq]

Got it working myself! The configuration file was fine. The thing that was missing was execute parameters.
For anyone who comes here with a similliar problem, make sure that your execute params are this:

As for journalbeat, I haven’t gotten to that yet. But I can sense a pattern here, these log collectors are pretty much the same (configuration wise), so you really don’t need to fiddle with them for hours, you can just look at the default config and deduce what should be changed.

For anyone interested:
Working configs for auditbeat and journalbeat:

Auditbeat:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
output.logstash:
   hosts: ["x.x.x.x:5044"]
path:                                                               
  data: /var/lib/graylog-sidecar/collectors/auditbeat/data      
  logs: /var/lib/graylog-sidecar/collectors/auditbeat/log                      
tags: 
    - linux                                                     
    - auditbeat                                                     
auditbeat.modules:
- module: file_integrity
  paths:
  - /bin
  - /usr/bin
  - /sbin
  - /usr/sbin
  - /etc
  - /var/log
- module: system
  datasets:
    - host      # General host information, e.g. uptime, IPs
    - process   # Started and stopped processes
    - login   
    - process
    - socket
    - user
    
state.period: 12s
user.detect_password_changes: true                                  #added
#processors:                                                        #Commented out
#  - add_host_metadata: ~                                           
#  - add_cloud_metadata: ~                                          

Journalbeat:

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

output.logstash:
    hosts: ["x.x.x.x:5044"]

path:                                                               
  data: /var/lib/graylog-sidecar/collectors/journalbeat/data      
  logs: /var/lib/graylog-sidecar/collectors/journalbeat/log
  
journalbeat.inputs:
- paths: []
  seek: head

[gsmith]

@cesq

Thank you for posting your resolve its much appreciate