Not seeing any messages from IIS (nxlog)

Graylog Central (peer support)
1

1. Describe your incident:

We have nxlog outputting IIS logs from multiple webservers to a GELF UDP input on our graylog node.
We had this configuration before, several months ago, with no issues. But now, while I am seeing network activity on the input, I am not seeing any messages in the database from that input from my production servers - I see messages from the pre-prod servers but not production. I’m not sure where I should be looking first to diagnose this problem.

2. Describe your environment:
graylog with a single node + elasticsearch on a separate server.

  • OS Information:

graylog+elasticsearch: Ubuntu (fossy)
Windows Server 2019

  • Package Version:

graylog 4.3.5
elasticsearch 7.10

  • Service logs, configurations, and environment variables:

not sure which logs are needed in this case, please let me know

3. What steps have you already taken to try and solve the problem?

I’ve verified that the nxlog conf is pointing to the correct server IP and ports for the two inputs.

4. How can the community help?

I’m at a bit of a loss and could use some direction on where to begin

2

Hello @factotum

I would check Graylog log files first, perhaps tailing them for errors or warnings.
Next, Elasticsearch logs looking for any clues.
Check out Nxlog log file on the IIS server.

Showing Nxlog configuration and the input Config would help and/or any logs that pertain to this issue would be good. Please when posting configuration use the markdown in the text box, :+1:

On Graylog you could use tcpdump and see is logs are making it to the Graylog server from that device.

3

Thanks for the response!

The tail end of my /var/log/graylog-server/server.log has many lines as follows (actual hostname replaced with servername.domain for confidentiality):

...[20]: index [graylog_0], type [_doc], id [2a2fad82-28b6-11ed-88a5-0050568bf8a1], message [ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse field [cs-host] of type [long] in document with id '2a2fad82-28b6-11ed-88a5-0050568bf8a1'. Preview of field's value: 'servername.domain']]; nested: ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=For input string: "servername.domain"]];]

Disclaimer: I am a newb at this. There was another guy who did most of the setup for the previous graylog instance, with me mostly observing over his shoulder.

But the above error suggests to me either that the nxlog conf isn’t correct, or that there was someplace else in the elasticsearch or graylog where I was supposed to define the input format.

I’m using NXLog CE 3.0.228
Here’s the nxlog conf formatting section:

#######################################################################
####                         IIS NXLOG                           ######
#######################################################################

<Extension w3c>
    Module	xm_csv
    Fields	$date, $time, $s-sitename, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs_User_Agent, $cs_Cookie, $cs_Referer, $cs-host, $sc-status, $sc-substatus, $sc-win32-status, $sc-bytes, $cs-bytes, $time-taken, $x-forwarded-for,
    FieldTypes	string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string, string
    Delimiter 		' '
	EscapeChar		'"'
    QuoteChar 		'"'
    EscapeControl 	FALSE
    UndefValue 		-
</Extension>

Does that appear to be the standard/default IIS log format?

Since (as far as I know) I’m using the same nxlog conf which worked before, could the format of the IIS logs have changed?

EDIT: I realized I forgot to add the Input config you asked for:

Editing Input Production Environment IIS Logs

Global
Should this input start on all nodes
Title
Production Environment IIS Logs
Bind address
0.0.0.0
Address to listen on. For example 0.0.0.0 or 127.0.0.1.
Port
12203
Port to listen on.
Receive Buffer Size (optional)
262144
The size in bytes of the recvBufferSize for network connections to this input.
No. of worker threads (optional)
16
Number of worker threads processing network connections for this input.
Override source (optional)
The source is a hostname derived from the received packet by default. Set this if you want to override it with a custom string.
Decompressed size limit (optional)
8388608
The maximum number of bytes after decompression.
4

Hello,

Oh I see, yeah this might be a little hard to troubleshoot. As for the configuration file I haven’t worked with Module xm_csv. This looks like an enterprise Edition ( paid version)for Nxlog. And IIS is in binary logging format.

As for your logs I seen this…

[cs-host] of type [long] in document with id ‘2a2fad82-28b6-11ed-88a5-0050568bf8a1’. Preview of field’s value: ‘servername.domain’]]

This shows me that you have a field called cs-host and its type is long, so this is a mapping issue. Looks like it should be a string. I also see this is on a INDEX graylog_0 I believe that is default and this is a new index. If this has been around for a while try rotating the index manually.
Navigate to System/Indices then click on Default index set. Under maintenance “drop down” click on “Rotate Active write index”. This should rotate you index. See if that clears it up.

image

To be honest I would look at the documentation for Nxlog using csv, since you stated that this was someone else’s configuration try to start simple. This may be where the issue is.

Maybe try a Raw/PlainText input using UDP on graylog , this might require the OUTPUT of Nxlog to be configured.

5

Ah, for this part, as I understand it’s just reading the W3C standard web log files output from IIS - it’s a delimited text log file same as you would probably get from Apache (but probably with a few extra fields tacked on to the end)

Thanks. If I understand you correctly, it sounds like the existing index was generated with a particular structure and I basically need to clear out that structure - is that right?

Would simply creating a new default index serve the same purpose?

6

True

Creating a new INDEX template would be better, But create a new index and attaching to a stream might work.

7

okay, great - this worked.

I created a new index set and made it the default index set.

Then I turned off my input as a precaution and rotated the active write index on the original default index set.

I waited till the index ranges were recalculated (and there were no documents in the active write index.)

Then I made the original default index set the default index set again, and I turned my input from the production servers back on.

It seems the nxlog conf from the non-production servers must have had an error or difference I didn’t catch.

Thanks again for your assistance.

8

Awesome and no problem, glad you were able to resolve this issue.
If you could mark this as resolved for future searches that would be great :+1:

1 Like
9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.