Indexing Failures

[OlympiaLady (Beth)]

I send my Windows server logs to Graylog using nxlog. After rebooting my Graylog server (one Debian node with Graylog/Elasticsearch/MongoDB) a few days ago, I have been getting indexing failures only for my IIS logs. It does not like a time field in the IIS log. I finally created a new stream and tied it to a new index. I set up a stream rule to have my IIS messages go into the stream, and now everything works. I would prefer to route my messages into the original streams and indexes, however. Otherwise, I have to fix all my dashboards and alerts.

It looks like one fix might be to create a custom Elasticsearch index template. Before doing that, however, I would like to know if there is any way to “fix” the existing index. My indexing failure is as follows:

{“type”:“mapper_parsing_exception”,“reason”:“failed to parse [time]”,“caused_by”:{“type”:“illegal_argument_exception”,“reason”:"Invalid format: “00:47:33” is malformed at “:47:33"”}}

Switching to Filebeat would work too, but there is quite a bit of work I would have to do to convert all my existing systems. I am trying to find the easiest solution. Any advice would be appreciated.

[jan (Jan Doberstein)]

I personal would create one processing rule that parse your given timestamp and translate that over to what is expected by Graylog/Elasticsearch.

That would look similar to:

 rule "parse_correct_date"
 when
 	has_field("timestamp") AND
 	to_string($message.source) == "windows_host"
 then
 	ts_orig = to_string($message.timestamp)

    // create new date object with correct timezone
    // pattern need to be adjusted
    let ts_new = parse_date(
        value: ts_orig, 
        pattern: "yyyy-MM-dd'T'HH:mm:ss.SSS", 
        timezone: "America/New_York");

    // set new timestamp with changed timezone 
    // the field will be written in the right way
    set_field("timestamp", ts_new);
 end

[OlympiaLady (Beth)]

Hi Jan,

Thank you for the advice. I will give it a try and post if it fixed my issue.

Beth