IIS logs into graylog

mr_m_cox · March 17, 2017, 3:33pm

Hi,
I have tried using nxlog on its own to get IIS logs into Graylog but that failed so I tried to bring SideCar into matters to get that to help but that seems to have only make it worse.

Please could someone explain or link to how I get IIS logs to appear in Graylog?

Thanks.

mr_m_cox · March 17, 2017, 4:16pm

I have decided to go bac kto basic and have uninstalled both nxlog and sidecar.

i then reinstalled nxlog and can get half the config working. The windows events are fine, its the IIS which isnt.

Here is my nxlog.conf

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel INFO

<Extension syslog>
    Module xm_syslog
</Extension>

<Extension gelf>
    Module xm_gelf
</Extension>

# Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.

<Extension w3c>
    Module xm_csv
    Fields $date, $time, $s-sitename, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes $time-taken
    FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, string, string, string
    Delimiter ' '
    QuoteChar   '"'
    EscapeControl FALSE
    UndefValue  -
</Extension>

# Enable json extension
<Extension json>
    Module      xm_json
</Extension>
 
# Convert the IIS logs to JSON and use the original event time
# Uncomment IIS_IN section if logging for IIS logging

<Input eventlog>
     # Use 'im_mseventlog' for Windows XP, 2000 and 2003
    Module      im_msvistalog
    # Uncomment the following to collect specific event logs only
    Query       <QueryList>\
                    <Query Id="0">\
                        <Select Path="Application">*</Select>\
                        <Select Path="System">*</Select>\
                    </Query>\
                </QueryList> 
</Input>

<Input IIS_Site1>
    Module im_file
    File "L:\\logs\\LogFiles\\W3SVC2\\u_ex*.log"
    SavePos TRUE
    Exec if $raw_event =~ /^#/ drop();    \
        else    \
        {    \
            w3c->parse_csv();    \
            $EventTime = parsedate($date + " " + $time);    \
            $SourceName = "IIS";    \
            $Message = to_json();    \
        }
</Input>


<Output Event_Out>
    Module      om_udp
    Host        10.10.0.79
    Port        12201      
    OutputType  GELF
</Output>

<Output IIS_Out>
    Module      om_udp
    Host        10.10.0.79 
    Port        5555
	
</Output>


<Route IIS>
     Path IIS_Site1 => IIS_Out
</Route>

<Route Events>
    Path eventlog => Event_Out
</Route>

mr_m_cox · March 17, 2017, 4:18pm

This is the input I have created using the RAW/PlainText UDP input type

Is this the right type of input?

Thanks.

harryw · March 21, 2017, 2:11am

You don’t need a separate input for the IIS logs, use the same as the Windows Event logs.
I route all my Windows Events and file logs using this one GELF UDP input.

Regards,

Harry W.

harryw · March 21, 2017, 2:13am

Also you can use the ‘nxlog snippet’ section to define the IIS log conversion …

{{if .Windows}}
<Extension w3c>
    Module xm_csv
    Fields $date, $time, $s-sitename, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $csUser-Agent, $csCookie, $cs-Referer, $cs-host, $sc-status, $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes, $time-taken, $x-forwarded-for
    FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, string
    Delimiter ' '
    QuoteChar '"'
    EscapeControl FALSE
    UndefValue -
</Extension>
{{end}}

That way all the nxlog configuration is in ‘graylog’ and you don’t have to configure the nxlog agent manually.

mr_m_cox · March 22, 2017, 10:15am

Thanks for teh help Harry, much apprecaited.

So now I have

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension gelf>
    Module      xm_gelf
</Extension>
 
<Input in>
    # Use 'im_mseventlog' for Windows XP and 2003
    Module      im_msvistalog
   Exec if ($EventID == 4202 or $EventID == 4208 or $EventID == 4302 or $EventID == 4304 or $EventID == 5004) drop();\
   else{\
        if ( $EventType == "INFO" ) $SyslogSeverityValue = 6;\
        if ( $EventType == "WARNING" ) $SyslogSeverityValue = 4;\
        if ( $EventType == "ERROR" ) $SyslogSeverityValue = 3;\
    }
</Input>

<Input IIS_Site1>
    Module im_file
    File "L:\\logs\\LogFiles\\W3SVC2\\u_ex*.log"
    SavePos TRUE
    Exec if $raw_event =~ /^#/ drop();    \
        else    \
        {    \
            w3c->parse_csv();    \
            $EventTime = parsedate($date + " " + $time);    \
            $SourceName = "IIS";    \
            $Message = to_json();    \
        }
</Input>
 
<Output out>
    Module      om_udp
    Host        10.10.0.79
    Port        12201
    OutputType  GELF
</Output>
 
<Route 1>
    Path        in, IIS_Site1 => out
</Route>

mr_m_cox · March 22, 2017, 10:15am

Where do I put in the code you suggested with the fields etc?

As you can probably tell, this is my first stab at graylog/nxlog so learning as I go.

Thanks.

mr_m_cox · March 22, 2017, 12:18pm

Finally got it working! Thanks to all that helped me out.

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension gelf>
	Module 		xm_gelf
</Extension>

<Extension fileop>
	Module 		xm_fileop
</Extension>

 <Extension json>
    Module      xm_json
</Extension>

Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c>
    Module 			xm_csv
    Fields 			$date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken
    FieldTypes 		string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer
    Delimiter 		' '
    QuoteChar 		'"'
    EscapeControl 	FALSE
    UndefValue 		-
</Extension>

<Input iis>
    Module    	im_file
    File    	"L:\\logs\\LogFiles\\W3SVC2\\u_ex*"
    SavePos  	TRUE

	Exec		if $raw_event =~ /^#/ drop();                    \
                else                                             \
                {                                                \
                    w3c->parse_csv();                            \
                    $EventTime = parsedate($date + " " + $time); \
                    $SourceName = "IIS";    					 \
                    $Message = to_json();    					 \
                }
</Input>

<Input eventlog>
    Module      im_msvistalog
</Input>

<Output graylog>
    Module      om_udp
    Host        xxx
    Port        12201
    OutputType	GELF

    #Use the following line for debugging (uncomment the fileop extension above as well)
	#Exec file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log", $raw_event);
</Output>

<Route eventlog>
   Path        eventlog => graylog
</Route>

<Route iis-to-graylog>
	Path		iis => graylog
</Route>

Nicodd17 · March 26, 2017, 4:24am

Thanks for share the solution!
I was having the same problem and now I have it working.
Cheers!

grayloglearn · June 7, 2017, 4:29am

Hi Team i have done same what you proceed with configuration. But i am not able get the logs. I have done below steps and this is my config file for iis logs.

Step 1) Created the input for GELF_UDP with the port 12201
Step 2 )changes has done in config file as follows below.

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/docs/

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Extension gelf>
	Module xm_gelf
</Extension>

<Extension fileop>
	Module 		xm_fileop
</Extension>

 <Extension json>
    Module      xm_json
</Extension>

#Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c>
    Module 			xm_csv
    Fields 			$date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken
    FieldTypes 		string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer
    Delimiter 		' '
    QuoteChar 		'"'
    EscapeControl 	FALSE
    UndefValue 		-
</Extension>

<Input iis>
    Module    	im_file
    File    	"L:\\logs\\LogFiles\\W3SVC2\\u_ex*"
    SavePos  	TRUE

	Exec		if $raw_event =~ /^#/ drop();                    \
                else                                             \
                {                                                \
                    w3c->parse_csv();                            \
                    $EventTime = parsedate($date + " " + $time); \
                    $SourceName = "IIS";    					 \
                    $Message = to_json();    					 \
                }
</Input>

<Input in>
    Module      im_msvistalog
# For windows 2003 and earlier use the following:
#   Module      im_mseventlog
</Input>



#<Output out>
#    Module      om_tcp
#    Host        192.168.1.1
#    Port        514
#    Exec        to_syslog_snare();
#</Output>

<Output out>
	Module om_udp
	Host X.X.X.X
	Port 12201
	OutputType GELF_UDP
</Output>

<Route 1>
    Path        in => out
</Route>

<Route iis-to-graylog>
	Path		iis => graylog
</Route>

But i am not able to get the IIS Can anybody help on this.

mr_m_cox · June 7, 2017, 8:20am

Hi,

Can you reformat that into a code block, its super difficult to read and spot the error as it is currently.

Thanks

grayloglearn · June 7, 2017, 8:55am

Hi mr_m_cox

Can you help how to do reformat that into a code block. I could not find it.

grayloglearn · June 7, 2017, 9:31am

define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension gelf>
	Module 		xm_gelf
</Extension>

<Extension fileop>
	Module 		xm_fileop
</Extension>

 <Extension json>
    Module      xm_json
</Extension>

Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c>
    Module 			xm_csv
    Fields 			$date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken
    FieldTypes 		string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer
    Delimiter 		' '
    QuoteChar 		'"'
    EscapeControl 	FALSE
    UndefValue 		-
</Extension>

<Input iis>
    Module    	im_file
    File    	"L:\\logs\\LogFiles\\W3SVC2\\u_ex*"
    SavePos  	TRUE

	Exec		if $raw_event =~ /^#/ drop();                    \
                else                                             \
                {                                                \
                    w3c->parse_csv();                            \
                    $EventTime = parsedate($date + " " + $time); \
                    $SourceName = "IIS";    					 \
                    $Message = to_json();    					 \
                }
</Input>

<Input eventlog>
    Module      im_msvistalog
</Input>

<Output graylog>
    Module      om_udp
    Host        xxxx
    Port        12201
    OutputType	GELF

    #Use the following line for debugging (uncomment the fileop extension above as well)
	#Exec file_write("C:\\Program Files (x86)\\nxlog\\data\\nxlog_output.log", $raw_event);
</Output>

<Route eventlog>
   Path        eventlog => graylog
</Route>

<Route iis-to-graylog>
	Path		iis => graylog
</Route>

My default logs path in C drive:/logs/w3cv/

micsnare · June 8, 2017, 11:45am

Hi @grayloglearn

can you post your nxlog.log from the log folder? this might give you an insight what isn’t working and why it isn’t starting.

best regards,
theresa

grayloglearn · June 8, 2017, 11:52am

Hi Micsnare,

Here is the nxlog.log

2017-06-07 11:20:54 INFO nxlog-ce-2.9.1716 started
2017-06-07 11:36:27 WARNING stopping nxlog service
2017-06-07 11:36:27 WARNING nxlog-ce received a termination request signal, exiting...
2017-06-07 11:36:28 ERROR module 'iis' is not declared at C:\Program Files (x86)\nxlog\conf\nxlog.conf:64
2017-06-07 11:36:28 ERROR module 'graylog' is not declared at C:\Program Files (x86)\nxlog\conf\nxlog.conf:64
2017-06-07 11:36:28 ERROR route iis-to-graylog is not functional without input modules, ignored at C:\Program Files (x86)\nxlog\conf\nxlog.conf:64
2017-06-07 11:36:28 INFO nxlog-ce-2.9.1716 started
2017-06-08 17:05:55 WARNING stopping nxlog service
2017-06-08 17:05:55 WARNING nxlog-ce received a termination request signal, exiting...
2017-06-08 17:05:56 ERROR module 'iis' is not declared at C:\Program Files (x86)\nxlog\conf\nxlog.conf:64
2017-06-08 17:05:56 ERROR module 'graylog' is not declared at C:\Program Files (x86)\nxlog\conf\nxlog.conf:64
2017-06-08 17:05:56 ERROR route iis-to-graylog is not functional without input modules, ignored at C:\Program Files (x86)\nxlog\conf\nxlog.conf:64
2017-06-08 17:05:56 INFO nxlog-ce-2.9.1716 started

grayloglearn · June 9, 2017, 5:56am

Hi,

I sent the nxlog.conf can you help me on this.

grayloglearn · June 9, 2017, 12:56pm

Hi team i am getting the below screen shot after installing the collector-side car in windows.
I have installed nxlog and sidecar in windows.

can anyone help onthis.

mr_m_cox · June 9, 2017, 1:09pm

You dont need anything to do with sidecars to get your IIS log into Graylog, just NXlog will do the job.

grayloglearn · June 12, 2017, 4:41am

Dear Matt,

Its good to hear with out sidecar we can get the IIS logs can you please help with step by procedure. I did what can i do but no luck. Can you please help where i did mistake. I have followed the below steps.

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/docs/

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Extension gelf>
	Module xm_gelf
</Extension>

<Extension fileop>
	Module 		xm_fileop
</Extension>

 <Extension json>
    Module      xm_json
</Extension>

#Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c>
    Module 			xm_csv
    Fields 			$date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken
    FieldTypes 		string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer
    Delimiter 		' '
    QuoteChar 		'"'
    EscapeControl 	FALSE
    UndefValue 		-
</Extension>

<Input iis>
    Module    	im_file
   File    	"C:\\inetpub\\logs\\LogFiles\\u_ex*.log"
    SavePos  	TRUE

	Exec		if $raw_event =~ /^#/ drop();                    \
                else                                             \
                {                                                \
                    w3c->parse_csv();                            \
                    $EventTime = parsedate($date + " " + $time); \
                    $SourceName = "IIS";    					 \
                    $Message = to_json();    					 \
                }
</Input>

<Input in>
    Module      im_msvistalog
# For windows 2003 and earlier use the following:
#   Module      im_mseventlog
</Input>



#<Output out>
#    Module      om_tcp
#    Host        192.168.1.1
#    Port        514
#    Exec        to_syslog_snare();
#</Output>

<Output out>
	Module om_udp
	Host X.X.X.X
	Port 12201
	OutputType GELF_UDP
</Output>

<Route 1>
    Path        in => out
</Route>

<Route iis-to-graylog>
	Path		iis => graylog
</Route>

drsockmonkee · June 16, 2017, 4:58pm

Glad to have found this thread. I am close but not successful yet.
I have a Windows 2012 (test server) and want to get the logs into the system.

I started over and downloaded a Ubuntu virtual machine as outlinned in the docs to run GrayLog. (it is running)
I downloaded and installed NXLog onto my Windows Test Server, and started a service with pointer to my log files ‘u_ex*.log’ from the inside of my Windows Test Server.

I have not done anything with: SideCar, and cannot get a HTTP message to show up on the VM.

I am sure that I am close but not there .

If anyone sees something wrong with the setup I am attempting, or the information below I would welcome input.

Andy…

MyConf file from the TestServer

define ROOT C:\Program Files (x86)\nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension gelf>
	Module xm_gelf
</Extension>

<Extension fileop>
	Module xm_fileop
</Extension>

<Extension _json>
Module      xm_json
</Extension>

<Extension w3c>
Module xm_csv
Fields $date, $time, $s-sitename, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes $time-taken
FieldTypes string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, integer, integer, integer, string, string, string
Delimiter ' '
QuoteChar   '"'
EscapeControl FALSE
UndefValue  -
</Extension>

<Input eventlog>
	Module im_msvistalog
</Input>

<Input iis>
	Module im_file
	File "C:\\inetpub\\logs\\testFolder\\u_ex*.log"	
	SavePos TRUE
	 Exec if $raw_event =~ /^#/ drop();    \
    else    \
    {    \
        w3c->parse_csv();    \
        $EventTime = parsedate($date + " " + $time);    \
        $SourceName = "IIS";    \
        $Message = to_json();    \
    }
</Input>

<Output graylog>
Module      om_tcp
Host        xx.xx.xx.xx  #my Graylog VM
Port        12201
OutputType	GELF_TCP
</Output>

<Route eventlog>
	Path eventlog => graylog
</Route>

<Route iis-to-graylog>
	Path iis => graylog
</Route>

My Log from ‘nxlog.log’

2017-06-16 11:33:09 WARNING stopping nxlog service
2017-06-16 11:33:09 WARNING nxlog-ce received a termination request signal, exiting...
2017-06-16 11:33:10 INFO connecting to xx.xx.xx.xx:12201
2017-06-16 11:33:10 INFO nxlog-ce-2.9.1716 started

My Source on the instance of GrayLog I am running on the downloaded VM I am running.

Topic		Replies	Views
Graylog shows eventlog from a machine but not IIS logs Graylog Central (peer support) sidecar , nxlog , nodatanx	4	1143	December 24, 2018
Graylog Sidecar and static windows server .log files Graylog Central (peer support) sidecar , nxlog , nodatanx	2	1413	June 29, 2017
61 / 5000 Resultados de traducción I need help sending IIS events to graylog with NXLog Graylog Central (peer support) sidecar , nxlog , nodatanx , send-csv-logsnx	2	708	August 17, 2021
How to get log from windows to graylog2.0.3 Graylog Central (peer support) sidecar , nxlog , nodatanx	8	8052	June 23, 2017
Nxlog file input Graylog Central (peer support) sidecar , nxlog , nodatanx	2	2592	April 12, 2018

IIS logs into graylog

Related topics