I was using nxlog to send windows and iis logs to Graylog successfully for about 2 years. I rebooted my Graylog server after some updates, and now all my IIS servers are not sending logs. I am getting errors relating to parsing time. So, I decided to try to use the Sidecar with Filebeat to get my IIS logs into Graylog. I am using the collector_sidecar_installer_0.1.7-1.exe installer, which seems to bundle Filebeat 6.4.1.
Filebeat is picking up the logs and sending them to Graylog, but they are not nicely parsed the way nxlog used to do it. I read on the Filebeat site that there is an IIS module. I added the following Beats snippet:
I could not do add the iis module without configuring filebeat.config.modules section. When I tried to add some additional options under -module: iis, the filebeat would not start.
Does anybody know how to get filebeat to actually parse the IIS logs?
I don’t “do Windows”, but as far as I remember, the Filebeat IIS module may return data in a way that isn’t picked up on by Graylog - I found that just setting up the sidecar to return the log file “as-is” and doing some stream rules on the “type” field (and perhaps additional fields) to classify stuff into a stream and then attaching a pipeline to it to handle the parsing/processing tends to work out better for a few reasons:
If Beats module changes, you don’t have a sudden “where’d all my stuff go” moment
You can parse out only the bits you need on the Graylog side
Simplifies the configuration for sidecars
And last but not least; it separates responsibility. Let Filebeat do the thing it’s good at (shipping logs from point A to point B), and let Graylog do what it is good at, instead of having a somewhat awkward mix where you end up with Filebeat doing some of it and Graylog doing the rest. Will save you management headache in the future.
just to clarify with the current stable sidecar/filebeat compo you can’t add the modules like elastic has thought off you should - when filebeat is configured via sidecar.
Currently that is only possible with filebeat vanilla - in 3.0 with the improved sidecar this will be possible but for now. Sorry.
You must keep in mind, nxlog and filebeat are different tools from different vendors so it is natural that the result is something different. You might want to work with the result and use the power of Graylog to normalize the data.
As you did not share how your logfiles are used to look with nxlog and how they look now with filebeat we are not able to help you with that process. But as @benvanstaveren already wrote - use the tools for what they are designed for and do not try to bend them.
Thank you jan and @benvanstaveren for your replies. I am not real good at the grok patterns and pipeline processing, but will give it a try. Below is an example message coming from IIS. If either of you know the quick fix to this, it would be appreciated.
The fields are:
date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
For those that are interested, I got a GROK pattern to work on the filebeat input. Note that it has to be tweaked depending on which fields you are logging.
