I wonder if someone can point me in the right direction please. I’ve trawled through the documentation, searched the forums and searched for tutorials but can’t seem to get this straight.
Basically the first hurdle is that we have log files on several servers (Windows/Linux) for different applications (IIS, Apache, Nginx, Postgresql, custom applications). I am struggling to visualise and understand how I should setup the filebeats input to handle the different combination of logs from each server.
For example, at the moment I have Server A sending in the postgresql logs. So if I wanted to collect only Postgresql logs from Servers A, B and C, they could all send to this one input and I would use the pipeline and streams to separate/combine the logs as required. But then what happens when I need to collect Postgresql and Apache logs from Server D? Do I create a new input specifically for that combination? So each server has its own side-car and configuration?
Is this the best/recommended way or is there a different way that I am not thinking about or come across.
Current versions installed:
Server: Graylog v3.1.3+cda805f
Sidecar: 1.0.2 on Ubuntu 18.04
Have the sidecar configuration tagged for the logs you are collecting in that group of servers and let the pipeline break things out based on those tags.
rule "RA-OWA-Access"
when
has_field("filebeat_tags") &&
contains(to_string($message.filebeat_tags),"OWA")
then
...
I was poking around with this some more - say you want separate tags for each log file area you are watching within a single collector configuration. In this case you will want to add a unique field (unique_log_tag: OWA) … as shown below:
filebeat:
inputs:
- type: log
enabled: true
# find owa logon and logoff but not when the HealthMailbox does it.
include_lines: ['auth.owa', 'logoff.owa', 'ClientDisconnect']
exclude_lines: ['HealthMailbox','^#']
fields:
unique_log_tag: OWA
ignore_older: 72h
paths:
- C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Owa\*.LOG
which would require a change in your rule to catch it:
rule "RA-OWA-Access"
when
has_field("filebeat_fields_unique_log_tag") &&
contains(to_string($message.filebeat_fields_unique_log_tag),"OWA")
then
Thanks @tmacgbay. I have managed to create the fields and I can see how that will work as far as getting the data to Graylog.
Unfortunately after a week and a half of reading, testing and failing, I am about ready to give up on Graylog. The biggest issue being the lack of complete examples of how to do a lot of things. e.g.
Managing sidecar configs for multiple servers or groups of servers
Examples of using Streams for different groups of logs, and for controlling user access to information
Seems grok patterns don’t support multiline fields in Graylog
Examples for processing CSV or any custom logs are non existent or very well hidden. Usually we get referred to the Marketplace, but then left high and dry if there’s nothing matching your file format in the market place (in my case, processing Postgresql and a custom NLog log file)
Full example of getting data through the pipeline and into the stream
As folders are not a feature for streams, there’s no guidance on how to organise streams for user access and ease of use.
Now these are just a few things I have personally struggled with. I may have missed the docs, but I’ve read through the official docs several times, searched the forums several times and done lots of online searching. Graylog seems like a very good and powerful system, but its adoption would be greater if the learning curve wasn’t so steep.
I’m also not complaining about a system that’s used by thousands and offered for free. Just feedback of my experience (granted, this is probably the wrong place anyway, but it’s written now.).
There is always the enterprise version - most of those things would be worked out… Opensource takes a lot of hand tooling and general maintenance… like a lot of things in life it’s Time vs. Money.
Our environment is mostly windows - happy to share more if you are continuing down the Graylog path.
