Timestamp converter fails when extra space present

Graylog Central (peer support)
1

I’m trying to extract a timestamp from a syslog message and use the date converter in order to set the timestamp with the correct timezone. An example of the full_message field is:

<130>: 2017 Aug  1 10:47:01.420 CST: %ACLLOG-2-ACLLOG_FLOW_INTERVAL: Src IP: 172.21.21.223, Dst IP: 172.21.83.168, Src Port: 5985, Dst Port: 61378, Src Intf: Ethernet101/1/1, Protocol: "TCP"(6), Hit-count = 3

Please note that there are 2 spaces between the month and day in these particular syslog messages.

The extractor is as follows:

{
  "id": "3cbabcb1-56c0-11e7-911b-de521aee9833",
  "title": "nexus timestamp",
  "type": "regex",
  "converters": [
    {
      "type": "date",
      "config": {
        "date_format": "yyyy MMM dd HH:mm:ss.SSS",
        "time_zone": "America/Regina"
      }
    }
  ],
  "order": 0,
  "exceptions": 0,
  "metrics": {
    "total": {
      "time": {
        "min": 4,
        "max": 140,
        "mean": 7,
        "std_dev": 5,
        "95th_percentile": 11,
        "98th_percentile": 13,
        "99th_percentile": 16
      },
      "rate": {
        "total": 1343149,
        "mean": 207.30197955354365,
        "one_minute": 158.00352660635258,
        "five_minute": 182.80628434011442,
        "fifteen_minute": 195.03650766965848
      },
      "duration_unit": "microseconds",
      "rate_unit": "events/second"
    },
    "condition": {
      "time": {
        "min": 0,
        "max": 6,
        "mean": 0,
        "std_dev": 0,
        "95th_percentile": 0,
        "98th_percentile": 0,
        "99th_percentile": 1
      },
      "rate": {
        "total": 1343149,
        "mean": 207.30192924147786,
        "one_minute": 158.0035266063527,
        "five_minute": 182.8062847700019,
        "fifteen_minute": 195.03650867751045
      },
      "duration_unit": "microseconds",
      "rate_unit": "events/second"
    },
    "execution": {
      "time": {
        "min": 0,
        "max": 14,
        "mean": 3,
        "std_dev": 0,
        "95th_percentile": 5,
        "98th_percentile": 5,
        "99th_percentile": 6
      },
      "rate": {
        "total": 1343149,
        "mean": 207.30188016684366,
        "one_minute": 158.00352660635258,
        "five_minute": 182.80628434387717,
        "fifteen_minute": 195.0365079212778
      },
      "duration_unit": "microseconds",
      "rate_unit": "events/second"
    },
    "converters": {
      "time": {
        "min": 96,
        "max": 195252,
        "mean": 200,
        "std_dev": 44,
        "95th_percentile": 278,
        "98th_percentile": 278,
        "99th_percentile": 278
      },
      "rate": {
        "total": 877,
        "mean": 0.13535632879978982,
        "one_minute": 0.1359173057640486,
        "five_minute": 0.12437532043255115,
        "fifteen_minute": 0.12511203024069864
      },
      "duration_unit": "microseconds",
      "rate_unit": "events/second"
    },
    "condition_hits": 0,
    "condition_misses": 0
  },
  "cursor_strategy": "copy",
  "source_field": "full_message",
  "target_field": "timestamp",
  "extractor_config": {
    "regex_value": "(\\s\\d{4}\\s[JFMASOND]{1}[a-z]{2,3}\\s+\\d{1,2}\\s\\d{2}:\\d{2}:\\d{2}\\.\\d{3})\\s[A-Z]{3,4}"
  },
  "creator_user_id": "si6986",
  "condition_type": "none",
  "condition_value": "",
  "converter_exceptions": 0
}

And the error I’m seeing in the graylog-server.log file is:

2017-08-01 10:47:01,420 ERROR: org.graylog2.plugin.inputs.Extractor - Could not apply converter [date] of extractor [3cbabcb1-56c0-11e7-911b-de521aee9833].
java.lang.IllegalArgumentException: Invalid format: "2017 Aug  1 10:47:01.420" is malformed at " 1 10:47:01.420"
        at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:945) ~[graylog.jar:?]
        at org.joda.time.DateTime.parse(DateTime.java:160) ~[graylog.jar:?]
        at org.graylog2.inputs.converters.DateConverter.convert(DateConverter.java:59) ~[graylog.jar:?]
        at org.graylog2.plugin.inputs.Extractor.runConverters(Extractor.java:262) [graylog.jar:?]
        at org.graylog2.plugin.inputs.Extractor.runExtractor(Extractor.java:249) [graylog.jar:?]
        at org.graylog2.filters.ExtractorFilter.filter(ExtractorFilter.java:73) [graylog.jar:?]
        at org.graylog2.messageprocessors.MessageFilterChainProcessor.process(MessageFilterChainProcessor.java:100) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.handleMessage(ProcessBufferProcessor.java:119) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:105) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:82) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:45) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

It looks to me like the converter isn’t able to handle the extra space between the month and the day, but I don’t know how to fix that. Any help would be appreciated.

2

You have to use the correct date format pattern in the Date Converter. dd expects the day component to always have 2 digits, such as 03, 15, etc.

Try using a single d in your pattern and see http://www.joda.org/joda-time/key_format.html for details.

3

Thank you very much for the suggestion jochen. I tried changing the Date Converter to use d rather than dd for the day. However, I’m still getting the same error.

As a side note, I looked at Joda-Time – Java date and time API - Formatting and it indicates that:

Number: The minimum number of digits. Shorter numbers are zero-padded to this amount. Thus, “HH” might output “09” whereas “H” might output “9” (for the hour-of-day of 9 in the morning).

4

Are you supplying the second space as part of the Date Converter date_format. When I highlight between MMM and dd I only see one space.

Can you try

yyyy MMM  d HH:mm:ss.SSS
5

Changing the Date Converter date_format as @billmurrin suggested fixed the problem. Thanks!!

I’m a little concerned that once we get to a two-digit day (Ex. Aug 10) that we’re going to run into a similar problem. Once we start getting logs for Aug 10, the log messages will just have 1 space between the month and the day. An example from 31 July 2017 can be seen below:

<130>: 2017 Jul 31 07:58:46.033 CST: %ACLLOG-2-ACLLOG_FLOW_INTERVAL: Src IP: 172.21.21.223, Dst IP: 172.21.83.168, Src Port: 5985, Dst Port: 63419, Src Intf: Ethernet101/1/1, Protocol: “TCP”(6), Hit-count = 1

6

As expected, I started receiving errors again as soon as we moved into a 2-digit day as follows:

2017-08-10 09:04:17,989 ERROR: org.graylog2.plugin.inputs.Extractor - Could not apply converter [date] of extractor [3cbabcb1-56c0-11e7-911b-de521aee9833].
java.lang.IllegalArgumentException: Invalid format: “2017 Aug 10 09:04:17.990” is malformed at " 10 09:04:17.990"
at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:945) ~[graylog.jar:?]
at org.joda.time.DateTime.parse(DateTime.java:160) ~[graylog.jar:?]
at org.graylog2.inputs.converters.DateConverter.convert(DateConverter.java:59) ~[graylog.jar:?]
at org.graylog2.plugin.inputs.Extractor.runConverters(Extractor.java:262) [graylog.jar:?]
at org.graylog2.plugin.inputs.Extractor.runExtractor(Extractor.java:249) [graylog.jar:?]
at org.graylog2.filters.ExtractorFilter.filter(ExtractorFilter.java:73) [graylog.jar:?]
at org.graylog2.messageprocessors.MessageFilterChainProcessor.process(MessageFilterChainProcessor.java:100) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.handleMessage(ProcessBufferProcessor.java:119) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:105) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:82) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:45) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

In order to resolve this, I had to change the date converter back to using 1 space between “MMM” and “d” rather than 2 spaces. Any recommendations? Is there maybe a way to tell the date converter that there will be 1 or 2 spaces between the “MMM” and “d”?

7

You could try using Grok patterns to normalize the date string before trying to parse it.

8

Thanks for the suggestion! I’ll look at trying to do that. Would I have to use a Grok pattern extractor on the full message and then a Regular Expression extractor with date converter on that field?

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.