# Ingesting IIS/Exchange CSV **URL:** \ **Category:** Graylog Central (peer support)\ **Tags:** pipeline-rules\ **Created:** [January 31, 2020, 12:19am UTC](https://community.graylog.org/t/ingesting-iis-exchange-csv/13704 "2020-01-31T00:19:48Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![michael.hoskin](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@michael.hoskin](https://community.graylog.org/u/michael.hoskin)\ **Post date:** [January 31, 2020, 12:19am UTC](https://community.graylog.org/t/ingesting-iis-exchange-csv/13704/1 "2020-01-31T00:19:48Z")
Hi, We’re in the process of adding our sources to Graylog and I’m trying to configure a pipeline to extract the fields from incoming messages from Exchange and IIS logs, which are stored in CSV format and fed through a filebeat collector. I have the messages entering Graylog without issue, and I’ve configured tags on the logs so we distinguish types, but I’m not able to get the pipeline to extract the fields. Here’s the logic we’ve been using: Filebeat config: filebeat: inputs: - type: log enabled: true paths: - D:\LogFiles\IMAP\*.LOG fields: {log\_type: Exch\_IMAP} - type: log enabled: true paths: - D:\LogFiles\MessageTracking\*.LOG fields: {log\_type: Exch\_MessageTracking} - type: log enabled: true paths: - D:\LogFiles\Connectivity\*.LOG fields: {log\_type: Exch\_Connectivity} - type: log enabled: true paths: - D:\LogFiles\W3SVC\*\*.log fields: {log\_type: Exch\_IIS} I’ve then got 2 pipeline rules, one to recognise a tag and the other to run the grok pattern (I tried having them in a single rule but that also didn’t work). Stage 0 rule: rule “Exch\_W3SVC\_Logs\_Tags” when contains(to\_string($message.fields\_log\_type), “Exch\_IIS”, true) then end Stage 1 rule: rule “Exch\_W3SVC\_Logs\_Extract” when true then let mess = to\_string($message.message); let parsed = grok(pattern: “%{EXCH\_W3SVC1\_LOG}”, value: mess, only\_named\_captures: true ); set\_fields(parsed); end Thus far the rules don’t seem to be running against incoming message. Any thoughts would be appreciated. ---
**Author:** ![shoothub](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/shoothub/32/6412_2.png) [@shoothub](https://community.graylog.org/u/shoothub)\ **Post date:** [January 31, 2020, 8:55am UTC](https://community.graylog.org/t/ingesting-iis-exchange-csv/13704/2 "2020-01-31T08:55:55Z")
1. First check your processing order: [https://docs.graylog.org/en/3.1/pages/pipelines/stream\_connections.html#the-importance-of-message-processor-ordering](https://docs.graylog.org/en/3.1/pages/pipelines/stream_connections.html#the-importance-of-message-processor-ordering) [https://docs.graylog.org/en/3.1/pages/pipelines/usage.html#configure-the-message-processor](https://docs.graylog.org/en/3.1/pages/pipelines/usage.html#configure-the-message-processor) 2. Try to debug rules conditions using debug function: > let debug\_message = concat("Tag ", to\_string($message.fields\_log\_type)); > debug(debug\_message); Then check log file /var/log/graylog-server/server.log for debug output [https://docs.graylog.org/en/3.1/pages/pipelines/functions.html#debug](https://docs.graylog.org/en/3.1/pages/pipelines/functions.html#debug) ---
**Author:** ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)\ **Post date:** [January 31, 2020, 3:26pm UTC](https://community.graylog.org/t/ingesting-iis-exchange-csv/13704/3 "2020-01-31T15:26:42Z")
One thing to note - if you have grok’ed in a field name with a space (or some odd char) in it, `set_fields()` will fail without logging why (I put in a change request for it)…this is where `debug()` was helpful! I have been setting up exchange and iis to use `split()` rather than GROK if possible, it isn’t as neat as GROK but it reads well - here is our iis: ``` rule "winbeat-iis" when has_field("filebeat_fields_tag") && ends_with(to_string($message.filebeat_fields_tag), "_iis",true) then let splittraf = split(" ", to_string($message.message)); // set_field("date", splittraf[0]); // set_field("time", splittraf[1]); set_field("s_ip", splittraf[2]); set_field("cs_method", splittraf[3]); set_field("cs_uri_stem", splittraf[4]); set_field("cs_uri_query", splittraf[5]); set_field("s_port", splittraf[6]); set_field("cs_username", splittraf[7]); set_field("c_ip", splittraf[8]); set_field("cs_user_agent", splittraf[9]); set_field("cs_referer", splittraf[10]); set_field("sc_status", splittraf[11]); set_field("sc_substatus", splittraf[12]); set_field("sc_win32_status", splittraf[13]); // set_field("sc_bytes", splittraf[14]); // set_field("cs_bytes", splittraf[15]); set_field("time_taken", splittraf[16]); end ``` ---
**Author:** ![michael.hoskin](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@michael.hoskin](https://community.graylog.org/u/michael.hoskin)\ **Post date:** [January 31, 2020, 9:03pm UTC](https://community.graylog.org/t/ingesting-iis-exchange-csv/13704/4 "2020-01-31T21:03:43Z")
Hmm, do you mean that using a space as a delimiter in a grok pattern won’t work, or that having a space in the field name is what causes the issue? I’ve tested the grok pattern using “test with sample data” function which seemed to work fine. Here’s the pattern, for reference: (?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) %{DATA:Server\_IP} %{DATA:Method} %{DATA:URI\_Stem} %{DATA:URI\_Query} %{DATA:Server\_Port} %{DATA:Client\_Username} %{DATA:Client\_IP} %{DATA:Client\_UserAgent} %{DATA:Referrer} %{DATA:HTTP\_Status} %{DATA:Protocol\_Substatus} %{DATA:Win32\_Status} %{DATA:Time\_Taken} %{DATA:X-Forwarder-For}$ We’ve temporarily setup input extractors, but at the moment we’ve configured multiple kinds of logs to go to the same input, so I’m wondering whether an input extractor is the better method or whether using pipelines would scale better? ---
**Author:** ![tmacgbay](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/tmacgbay/32/3878_2.png) [@tmacgbay](https://community.graylog.org/u/tmacgbay)\ **Post date:** [January 31, 2020, 9:25pm UTC](https://community.graylog.org/t/ingesting-iis-exchange-csv/13704/5 "2020-01-31T21:25:27Z")
There are a series of special characters that set\_fields() will silently die on if they are in the field name that it’s trying to create. None of which are in your GROK. Extractors work fine and when I asked the preference in the forum, the answer was ambivalent. While I was troubleshooting a different issue I moved everything from extractors to pipelines. We have a small environment so it wasn’t an issue. So ignore me and go with shoothub’s checking the processing order and using `debug()` to see what is happening in the pipeline in the server logs ` tail -f /var/log/graylog-server/server.log` ---
**Author:** ![cbgraham](https://avatars.discourse-cdn.com/v4/letter/c/9f8e36/32.png) [@cbgraham](https://community.graylog.org/u/cbgraham)\ **Post date:** [February 7, 2020, 12:27am UTC](https://community.graylog.org/t/ingesting-iis-exchange-csv/13704/6 "2020-02-07T00:27:13Z")
I recently setup Exchange 2016 iis log ingestion. Here is my grok pattern. I don’t know if it’s ideal, but it works. `%{TIMESTAMP_ISO8601} %{IP:s-ip} %{NOTSPACE:cs-method} %{NOTSPACE:uri-stem} %{NOTSPACE:uri-query} %{BASE10NUM:port} %{NOTSPACE:username} %{IP:c-ip} %{NOTSPACE:user-agent} %{NOTSPACE:UNWANTED} %{BASE10NUM:status} %{NOTSPACE:UNWANTED} %{NOTSPACE:UNWANTED} %{BASE10NUM:time-taken} (-|%{IP:src_ip})` ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [February 21, 2020, 12:27am UTC](https://community.graylog.org/t/ingesting-iis-exchange-csv/13704/7 "2020-02-21T00:27:22Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.