Hello,
so far it exist 438219 ways to ship logs to graylog, but what is the recommended thing?
Eg: Apache you can use filebeat or you do it via nc, but what next? Use grok pattern or regexp to get the messages?
Or what is the best way to sip server logs, in the past we can use rsyslog, but since journald its more and more uncommon,or not? What I see is that can happen that journald dont push messages to rsyslog if there are many (like on mailserver),so should I use rsyslog, filebeat or may better journalbeat but if I use journalbeat, how can I get logfiles from apache? So far as I can see is that journalbeat doesn’t support files.
Or may I should use nxlog or Vector instead of beats, so what is the “best” way to ship logs to graylog (with in mind to have a small software footprint), and is grok better than regexp, etc…
I don’t know that you’re going to get an answer here that prescribes everything to you. The point of all of the options is so that you have the flexibility to assess and decide for yourself. Some of that is going to be dependent on the technology gaps and differences you’ve described, some of it will depend on your skillset and preferences, and some of it will be dictated by your business needs
For example, personally I abuse the syslog specification and configuration quite a lot to accommodate custom processes developed internally because I find it easier to capture exactly what I want that way. For me, it works. For others, maintaining custom configurations and processes is too much administrative overhead because they have 1,000 other systems to worry about.
If you’re having problems or challenges with specific configurations/options or data sources please do share them, there are many people on here with a variety of experience and knowledge that are willing to help. Don’t forget to check out the Graylog Forwarder too.
hi @ttsandrew ,
thanks for you answer.
Lets say I’m starting in a new envirment where I can do what I want / whats the best, and skills can be learned
First question would be the best way to ship logs to graylog, mostly on Linux (with systemd) but also some windows machines (~3), but i want a logshipper that works on linux, windows, can work with systemd and also with files and it should be ONE server not many…
We have several dozen servers shipping logs and literally all of our Graylog inputs are syslog. Most traffic arrives via a TCP input with TLS, but some traffic doesn’t support encryption on send so that is unfortunately clear text, and then we have a UDP input running to catch one offs like environmental sensors that don’t support TCP syslog.
For us this works. Most of our appliances natively support syslog and configuring syslog on *nix appliances is easy. The trick for us was windows clients. We have developed a few custom internal processes depending on the application need, generally a service is just formatting the data into syslog standard using an internally developed standardized lib and then ships it off to Graylog. If we have some specific need for parsing fields we make sure the data being shipped is parsable (for us JSON or comma delimited) and then we use extractors.
For a lot of people syslog would be a ridiculous standard because their payloads are more complex, larger, or their environments too complex in some other way.
For a mostly linux environment if you’re gathering standard application and system logs my money would be for syslog. Graylog doesn’t strictly enforce the standard so you can fudge field definitions if you need to.
ok but it happens several time that we dont get logs from rsyslog on our mailserver due the amount of messages. so we should use both.
But thanks to see how you use it
