We’ve started moving our logs, mostly syslog to graylog. It’s working great, and we want to take it further and add more structure to our log data.
Most of it is syslog, where we’re just shipping everything to graylog until we get a sense of what we can and need to store.It all goes to a single syslog input and to the default stream.
Apache logs are sent by filebeat to a single beats input. Then we have a netflow input for netflow data from routers and a GELF UDP input for our in-house java application.
Now, let’s say I want to create a dashboard for mail servers. Logs from our postfix mail servers is coming into the syslog input today and sent to the default stream. Where would it be best to extract data from postfix logs? On the host before sending to graylog, in an extractor (apparently being deprecated in favor of pipelines) or in a pipeline? If the answer is pipeline, should I first direct the messages to a dedicated stream and then run the pipeline there?
I know there are many ways to skin this cat, but I want to learn from others about faults and benefits of different ways.