Currently I have a setup that has syslog messages where the source is the MACADDRESS (ie 507E5DC79D8A) As I am using the SYSLOG codec this field is parsed over into the source field.
No problems there but if I try to create overviews (dashboards/widgets) for instance “Aggregating count() by source” with this field as parameter i get the following error:
While retrieving data for this widget, the following error(s) occurred:
When I select data having an IPaddress as the source field I get a correct count.
So I worked/looked already on a couple of solutions by reading others posts.
don’t use syslog but create a custom codec using GROK
try to create a pipeline to parse the source field in a separate “new” field
try to ignore the source and set the source with the $message.gl2_remote_ip
weirdly enough the pipeline didn’t work, so I starting to look back at what started the problem. Why can’t I somehow not index on this macaddress fields.
Has anybody any ideas, or suggestions to look into.
you might want to create a RAW Input on Graylog to ingest that logs to.
it looks like they are not following any kind of “normal” syslog dialect so that Graylog can parse it. When you see the raw message you can start parsing the content out yourself.
For that the processing pipeline might be the best option.
Thanks I can look into this indeed. I have btw also the control on the syslog client so maybe when I know which part is not conform standard I can look into that. What is the best approach to get this format validated, also by getting RAW? I think it is allowed to choose any value in the source field right? So a MAC should be no issue.
The pipeline didn’t work on the syslog messages, so let’s see how it works on the RAW input.
In my experience, there isn’t much configuration available on clients. You set a target, a port, and maybe a protocol. The most I have seen is be able to pick an RFC to use when sending.
If my client isn’t sending what I want, it needs a RAW input.
That’s for devices, it would different if you have access to an application that’s sending the data.
Closing the loop: So i finally managed to write a GROK pattern and create a pipeline for this RAW input. RAW input because next to source other fields where incorrect interpereted, like facility and severity. Apparently the format I get is “classic” SYSLOG which is not in-line with the existing build-in SYSLOG INPUT plugin. Also the order of the message-chain and pipeline was wrong which made all my pipelines not able to interact. I changed that in the config.
