Filebeat javascript processors

Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!

1. Describe your incident:
Hi, I’ve got 2 questions here!

  1. Is it possible to use filebeat processors with graylog sidecars?
  2. If it’s not, then how could I achieve some similar behavior?

In my case I just wanted to filter specific information from messages and convert them into separate fields so I could filter the logs easily by those fields…

Below is my test config for filebeat.

# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}

   hosts: ["${user.graylog_host}:5044"]
   data: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/data
   logs: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/log


- type: filestream
  id: snort-filestream
  enabled: true
    - /var/log/snort/alert_json.txt
    - /var/log/snort/appid-output.json
    - ndjson:
        target: "snort3"
        add_error_key: true
        overwrite_keys: true
    event_source_product: snort3

- type: filestream
  id: zeek-filestream
  enabled: true
    - /opt/zeek/logs/current
    - ndjson:
        target: "zeek"
        add_error_key: true
        overwrite_keys: true
    event_source_product: zeek
- type: filestream
  id: apache-filestream
  enabled: true
  - /var/log/apache2/access.log
  - /var/log/apache2/error.log
  - /var/log/httpd/access_log
  - /var/log/httpd/error_log
  fields_under_root: true
      event_source_product: apache_httpd
- type: filestream
  id: lastlogin-id
  enabled: true
    - /var/log/.lastlogin.log
    - script:
      lang: javascript
      source: >
        function process(event) {
            const message = event.Get("message");
            // Regular expression to match an IP address
            var ipRegex = /\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b/;
            // Match the IP address in the input string
            var match = message.match(ipRegex);
            // If a match is found, return the IP address
            if (match !== null) {
                event.Put("ip", match[0]);
            } else {
                event.Put("ip", "No IP address found");

When I remove the processors part of the config, graylog logs the .lastlogin files fine, but otherwhise, it just doesn’t log… Maybe it’s a javascript problem? :thinking:

2. Describe your environment:
Currently my setup is something like this…

  • OS Information: Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-172-generic x86_64)

  • Docker images:

    • mongo:5.0
    • opensearchproject/opensearch:2.4.0 (yes, I'm actually using OS instead of ES)
    • graylog/graylog:5.2.4-1
  • Package Version:

    • sidecar: 1.5.0
  • Service logs, configurations, and environment variables:

I’ll provide further in the chat as needed.

3. What steps have you already taken to try and solve the problem?
What I do know so far is that when I add the processor config on the .yml file, it stops to log, and when I remove it, it starts logging again…

4. How can the community help?

If the community has already done something like this (processing messages and creating new fields) it could be very helpful to have some guide on how to do it!

Anyway… I think I need some kind of guide or documentation instead of solving a specific problem…

Yes sidecar supports anything in the beats config standard, it is just delivering that yml file to the right location on the machine, but isn’t really doing anything to it.
So it will be an error of some kind in your config, they are very picky. I don’t know them well enough to help, but there isn’t anything sidecar specific with your issue so anything you read online about beats will apply.

Awesome! Well in that case I’ll take a look deeper in the beats page, thanks for the response! :+1:

Got the solution…

Indeed I’ve got 2 problems…

  • yml indentation

Solved by correcting the indentation

  • javascript code

Filebeat processes ECMAScript 5.1 code, so use a converter for your javascript code, I had some code that was not yet implemented on this version, after converting it everything went fine! :ok_hand:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.