I am attempting to enrich my linux logs that are sent to graylog via filebeat by way of sidecar. I configured rsyslog to also provide log level and facility as shown here:
I would like to create a pipeline rule (or rules) that extracts the first character - the log level - and the third - the facility - and stores them as log_level and facility, respectively. From there, I would use the contents of these fields against a look up so that, for instance, I would end up with:
log_level
6
log_name
INFO
facility_number
3
facility_name
system_daemons
Here was my first attempt that did jack poop:
rule "Extract Log Level"
when
to_string($message.log_file_path) == "/var/log/syslog"
OR to_string($message.log_file_path) == "/var/log/auth.log"
OR to_string($message.log_file_path) == "/var/log/kern.log"
OR to_string($message.log_file_path) == "/var/log/mail.log"
OR to_string($message.log_file_path) == "/var/log/cron.log"
then
let LogLevel = regex(
pattern: "(?<=^.{0}).{1}",
value: to_string($message.message)
);
end
And, honestly, I just frankensteined that from google-foo. I have little to no understanding of regex and pipelines.
So, as a secondary request, any recommendations to learn regex and pipeline logic? I prefer books (like Packt) to videos.
rule "Extract fields"
when
has_field("message")
then
let log = regex("(loglevel=\\s*(\\S+))", to_string($message.message));
set_field("log_level", log["0"]);
debug (log); <------ For debug purposes only
end
As for create multiple fields I would suggest using Key_value if possible.
Example of mine for Fortinet Firewall
rule "forti_keys_to_fields"
when
starts_with ((to_string($message.message)) , "type", true )
then
// use $ tail -f /var/log/graylog-server/server.log to watch for the results of the below debug message
debug(concat("++++++++++ Full Message: ", to_string($message.message)));
// extract all key-value from "message" and prefix it with auditd_
set_fields(
fields: key_value(
value: to_string($message.message),
trim_value_chars: "\"",
trim_key_chars:"",
delimiters:"|",
kv_delimiters:"="
),
prefix: "forti_"
);
end
I quite agree; I am not sure why the syslog didn’t include Level and Facility. This is on a ProxMox rig which is based on Debian 12. Journald was enabled instead of syslog. I had to enable via rsyslog with the following config:
#################
#### MODULES ####
#################
module(load="imuxsock") # provides support for local system logging
module(load="imklog") # provides kernel logging support
#module(load="immark") # provides --MARK-- message capability
module(load="imjournal")
module(load="mmjsonparse")
# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")
# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")
###########################
#### GLOBAL DIRECTIVES ####
###########################
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$template precise,"%syslogpriority%,%syslogfacility%,%timegenerated%,%HOSTNAME%,%syslogtag%,%msg%\n"
$ActionFileDefaultTemplate precise
