Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question.
Don’t forget to select tags to help index your topic!
1. Describe your incident:
Hi, I’ve got 2 questions here!
- Is it possible to use filebeat processors with graylog sidecars?
- If it’s not, then how could I achieve some similar behavior?
In my case I just wanted to filter specific information from messages and convert them into separate fields so I could filter the logs easily by those fields…
Below is my test config for filebeat.
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
output.logstash:
hosts: ["${user.graylog_host}:5044"]
path:
data: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/data
logs: ${sidecar.spoolDir!"/var/lib/graylog-sidecar/collectors/filebeat"}/log
filebeat.inputs:
- type: filestream
id: snort-filestream
enabled: true
paths:
- /var/log/snort/alert_json.txt
- /var/log/snort/appid-output.json
parsers:
- ndjson:
target: "snort3"
add_error_key: true
overwrite_keys: true
fields:
event_source_product: snort3
- type: filestream
id: zeek-filestream
enabled: true
paths:
- /opt/zeek/logs/current
parsers:
- ndjson:
target: "zeek"
add_error_key: true
overwrite_keys: true
fields:
event_source_product: zeek
- type: filestream
id: apache-filestream
enabled: true
paths:
- /var/log/apache2/access.log
- /var/log/apache2/error.log
- /var/log/httpd/access_log
- /var/log/httpd/error_log
fields_under_root: true
fields:
event_source_product: apache_httpd
- type: filestream
id: lastlogin-id
enabled: true
paths:
- /var/log/.lastlogin.log
processors:
- script:
lang: javascript
source: >
function process(event) {
const message = event.Get("message");
// Regular expression to match an IP address
var ipRegex = /\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b/;
// Match the IP address in the input string
var match = message.match(ipRegex);
// If a match is found, return the IP address
if (match !== null) {
event.Put("ip", match[0]);
} else {
event.Put("ip", "No IP address found");
}
}
When I remove the
processorspart of the config, graylog logs the.lastloginfiles fine, but otherwhise, it just doesn’t log… Maybe it’s a javascript problem?
2. Describe your environment:
Currently my setup is something like this…
-
OS Information:
Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-172-generic x86_64) -
Docker images:
mongo:5.0opensearchproject/opensearch:2.4.0 (yes, I'm actually using OS instead of ES)graylog/graylog:5.2.4-1
-
Package Version:
sidecar: 1.5.0
-
Service logs, configurations, and environment variables:
I’ll provide further in the chat as needed.
3. What steps have you already taken to try and solve the problem?
What I do know so far is that when I add the processor config on the .yml file, it stops to log, and when I remove it, it starts logging again…
4. How can the community help?
If the community has already done something like this (processing messages and creating new fields) it could be very helpful to have some guide on how to do it!
Anyway… I think I need some kind of guide or documentation instead of solving a specific problem…
