I am able to get logs from our MKT router if in BSD syslogs but only using a RAWudp with no log level. I am also able to receive logs when not using BSD I can receive logs all with a -1 level.
We are collecting simple (no BSD) logs. They are fed via separate input with two regex extractors. First extractor extracts topics part to new field topics, second extractor extracts log level from field topics to new field severity.
well to me it looks like you should check BSD syslog and have it come in as rawUDP. The seems to have the most information in it, including the facility and level. The tricky part will be extracting it.
According to RFC3164 the data between the <> is the priority field which consists of the the facility and severity of the message, in your case <46> indicates security/informational (4=security 6=informational) at this point, you can simply extract the PRI field with something like
<%{BASE10NUM:message_priority}>
which would result in
message_priority
46
Then you can use this like any other field and search or alert against it. If you want to split message_priority into the seperate facility and level, you would be better served doing all of this in a multistaged pipeline.
## Extract topics from message
Extract beginning of the 'message' into 'topics'
Extractor type: Regular expression
Source field: message
regex_value: (.+?) .*
Store as field: topics
## Extract severity from topics
Extract using keywords from 'topics' into 'severity'
Extractor type: Regular expression
Source field: topics
regex_value: (info|warning|error|critical)
Store as field: severity
Second one depends on first, so arrange them accordingly.
OK i have a solution More or less.
MKT settings
remote with BSD disabled
port 2514
Graylog
individual Input Raw on port 2514
extractor1 one for each level desired
Regular expression
Source field=message
Regular expression=^.*(info)
Field matches regular expression=info
Store as field=LOGLEVEL
extractor2 one for each level desired
Replace with regular expression
Source field=LOGLEVEL
Regular expression=info
Replacement=6
Store as field=level
This results in logs showing correct log level in graylog as well as in librenms.
I think i can make these extractors more efficient using pipelines as @cawfehman but could use some help. As this is a working solution i will back burner the pipeline unless I receive some unsolicited help(plz). Thank you @Karlis and @cawfehman for your time and effort.
O and to share something in return. Here is the script i have written to pass the remote log config to all 70 or so Mikrotics in our environment.
Glad to hear you got it working, that’s the most important part. Now you can tweak it, if necessary or needed. Good on you for giving back with the script as well. That’s good internet karma.
As for the pipeline, give it a go and ask questions if you get stuck… post as much of it as you feel comfortable with, and it always helps for those helping you to see a sample log message.
