ES fails to parse date/timestamp for Win IIS logs, logs not indexed

Graylog Central (peer support)
1

I am using nxlog to parse and ship Windows IIS logs to Graylog and noticing that when I create a simple stream to grab logs with a specific collector ID and try to send these logs to said stream, they do not arrive, stream shows no logs. When I check System >> Overview >> Indexer failures I see lots of errors like this:

ElasticsearchException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse field [time] of type [date] in document with id 'e1a4a471-28a2-11ed-97c8-00155d202636'. Preview of field's value: '20:32:10']]; nested: ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=failed to parse date field [20:32:10] with format [strict_date_optional_time||epoch_millis]]]; nested: ElasticsearchException[Elasticsearch exception [type=date_time_parse_exception, reason=Failed to parse with all enclosed parsers]];

When I pause the stream, the messages do begin to appear in the ā€˜All Messagesā€™ stream and the index failures go away. Then when I start the stream again, the logs fail to index, nothing shows up in the stream, and the index failures start again.

Seems to be something wrong with the ability to parse the date/timestamp.

From what I gather, ES expects them to be in format ā€˜strict_date_optional_timeā€™ with optional epoch timestamp:

date_optional_time or strict_date_optional_time
A generic ISO datetime parser, where the date must include the year at a minimum, and the time (separated by T), is optional. Examples: yyyy-MM-dd'T'HH:mm:ss.SSSZ or yyyy-MM-dd.

but IIS logs date/time look like this for example:

2021-10-30 21:59:35

How can I manipulate the date/timestamp on the client side before shipping to Graylog so that ES can properly parse it and index and the logs show up in my stream? Please let me know if you need to see portions of my configs etc.

2

Hello @big-tuna-28

This is odd, If All Message stream is fine and there is no index error/s I wonder what the issue is with the other stream.

Can I ask what are the configuration with the faulty stream?

3

Are ā€œAll Messagesā€ and your IIS-stream on the same index set? It might happen, that the timestamp-field is changed to some other type in the IIS-stream.

4

The stream just has one stream rule that looks for a specific string in the ā€˜collector_node_idā€™ field

5

They are on different index sets. All Messages stream is on the Default index set where our custom stream is on a new index set that we created. However I checked both index sets and they are configured the exact same way (ie. shards #, replicas #, rotation strategy, rotation period, etc.) they are identical.

You gave me an idea though - since the index sets are identical, I switched my custom stream to use the Default index set and everything works perfectly. But I need to be able to create custom indexes and have them working vs dumping everything into Default. And other question is why is the indexing/parsing of timestamp different on two index sets that are identical in configuration?

UPDATE: Based on the above I thought maybe it was an issue specific to my index set that I created. I created a brand new index set and migrated the stream to it and it seems to be working. I still canā€™t explain what is wrong with the first index set I created, but new one seems to be indexing without issue.

6

Awesome!! glad you resolved your issue :+1:

If you could mark this as resolved for future searches that would be great.

7

Elasticsearch Indexes guess at what a field is on the initial message that comes into the index. This means that if you are playing with the data as you start and a timestamp field accidentally has a string value, the index will consider it a string for all future messages. If you rotate the index (or create a new index) , it will again guess field type based on the data coming. You can also force a field type with a custom mappingā€¦ there are other posts on how to do thatā€¦

2 Likes
8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.