# Issues with date type converter format string

**URL:** <https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524>\
**Category:** Graylog Central (peer support)\
**Created:** [October 21, 2021, 9:49am UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524 "2021-10-21T09:49:57Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![aft2d](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/aft2d/32/8547_2.png) [@aft2d](https://community.graylog.org/u/aft2d)\
**Post date:** [October 21, 2021, 9:49am UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/1 "2021-10-21T09:49:57Z")

</div>

Hi there,

## Description of your problem

I’m receiving nginx error log messages on a syslog input and try to read the timestamp from the message.  
I created an extractor that extracts timestamp from the message:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/2/2af902b6364674aa052f05dfc259d36fb9163de9.png)  
Extractor works^^

Then I added a date-time converter and configured it like this:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/d/d4eaa4141d9ec4f235131ccfd6ed8546f0c204b8.png)

Unfortunately this always results in this error message:

```auto
Value <2021/10/21 09:35:16> caused exception: Invalid format: "2021/10/21 09:35:16" is malformed at "/10/21 09:35:16".

```

## Description of steps you’ve taken to attempt to solve the issue

After reading the [docs](https://docs.graylog.org/docs/extractors), I tried a bunch of other format strings like:  
yyyy//MM//dd HH:mm:ss  
yyyy’/MM’/dd HH:mm:ss  
but they always result in the same error message.

From all examples given in the docs and the placeholder in the graylog web UI I’d say that my first format string from the screenshot above should be correct. Am I missing something?

## Environmental information

### Operating system information

```auto
cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"

```

### Package versions

```auto
dpkg -l | grep -E ".*(elasticsearch|graylog|mongo).*"
ii elasticsearch-oss 7.10.2 amd64 Distributed RESTful search engine built for the cloud
ii graylog-4.2-repository 1-4 all Package to install Graylog 4.2 GPG key and repository
ii graylog-server 4.2.0-3 all Graylog server
ii mongodb-org 4.0.27 amd64 MongoDB open source document-oriented database system (metapackage)
ii mongodb-org-mongos 4.0.27 amd64 MongoDB sharded cluster query router
ii mongodb-org-server 4.0.27 amd64 MongoDB database server
ii mongodb-org-shell 4.0.27 amd64 MongoDB shell client
ii mongodb-org-tools 4.0.27 amd64 MongoDB tools

```

Logs don’t show anything useful, just messages that I’m updating the extractor settings 🙂

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![Winklerschulbedarf](https://avatars.discourse-cdn.com/v4/letter/w/ec9cab/32.png) [@Winklerschulbedarf](https://community.graylog.org/u/Winklerschulbedarf)\
**Post date:** [October 21, 2021, 10:00am UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/2 "2021-10-21T10:00:10Z")

</div>

Hello there!  
So i got the exact same problem after i updated to graylog 4.2.0  
Seems to be a problem with the update.

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [October 21, 2021, 9:48pm UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/3 "2021-10-21T21:48:04Z")

</div>

Hello

I noticed your Date Format string is `yyyy/MM/dd HH:mm:ss.`

> [@aft2d](#):
>
> `caused exception: Invalid format`

Not 100% sure but I think it might be from your forward slashes. Maybe try this configuration  
`yyyy-MM-dd HH:mm:ss.SSS`

Below worked in my lab.

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/8/8214c5716480104d0106bc3bedbea6a12453c910.png)

Hope that helps

---

<div class="post-metadata">

**Author:** ![aft2d](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/aft2d/32/8547_2.png) [@aft2d](https://community.graylog.org/u/aft2d)\
**Post date:** [October 22, 2021, 8:48am UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/4 "2021-10-22T08:48:36Z")

</div>

Hi,

I followed your suggestion and used: `yyy-MM-dd HH:mm:ss`  
Unfortunately with the same result.

I mean, I need to use the slashes? My input string is `2021/10/22 08:42:47` for example, and I can’t change it. That’s also why I need to use the Date converter.

---

<div class="post-metadata">

**Author:** ![Winklerschulbedarf](https://avatars.discourse-cdn.com/v4/letter/w/ec9cab/32.png) [@Winklerschulbedarf](https://community.graylog.org/u/Winklerschulbedarf)\
**Post date:** [October 22, 2021, 9:05am UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/5 "2021-10-22T09:05:25Z")

</div>

Sadly this didnt work out for me as well.  
I also tried to change the date thats delivered to graylog by my powershell script but nothing worked.  
My config is pretty much the same as the one from @aft2d.

---

<div class="post-metadata">

**Author:** ![Winklerschulbedarf](https://avatars.discourse-cdn.com/v4/letter/w/ec9cab/32.png) [@Winklerschulbedarf](https://community.graylog.org/u/Winklerschulbedarf)\
**Post date:** [October 22, 2021, 11:18am UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/6 "2021-10-22T11:18:49Z")

</div>

Ok so when trying to copy the needed timestamp from the message and convert it into another field i.e. timestamp2 everything works just fine.

![grafik](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/7/711eb991e622a5716060add31a88d9f3c9c0cb97.png)

You can notice the difference of the format between the two timestamps.  
Maybe this matters?

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [October 22, 2021, 9:15pm UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/7 "2021-10-22T21:15:51Z")

</div>

Hello,

> [@Winklerschulbedarf](#):
>
> convert it into another field i.e. timestamp2

How did you go about converting it? Pipeline?

If you have a field called “ **timestamp** ” already you can covert it via pipeline but I believe If you creating a regular expression you cant use the same field you have to call it something else. Perhaps that is where the issue is.

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [October 22, 2021, 11:00pm UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/8 "2021-10-22T23:00:30Z")

</div>

Hello,

Can you show by example the date/time from nginx message your trying to extract?

On another note if you created a new field for you Regex then instead of using a converter maybe a pipeline would help.

[https://docs.graylog.org/docs/functions#parse-date](https://docs.graylog.org/docs/functions#parse-date)

Maybe something like this.

```auto
rule “parse date”
when
has_field(“some_field”)
then
let new_date = parse_date(to_string($message.some_field), “yyyy-MM-dd’T’HH:mm:ssZZ”));
set_field(“install_datetime”, new_date);
end

```

Hope that helps

EDIT:

So I’ve been testing this in my lab, just so happens I have nginx installed on a server. Using you Regex configuration from the picture you posted.

`^.*:\s(\d\d\d\d/\d\d/\d\d\s\d\d:\d\d:\d\d)\s.*$`

It seams not to be working

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/0/04097b90ddfd77c78ee04ff85e54b119591b5815.png)

but this works

`^.*(\d\d\d\d/\d\d/\d\d\s\d\d:\d\d:\d\d)\s.*$`

EDIT:  
I see your issue now.

I’ve been testing this in my lab for a while and what I did was create a Syslog UDP input for my Nginx logs. Then I configured a regex extractor for testing both of the following configuration which gave me the same results.

```auto
^.*:\s(\d\d\d\d/\d\d/\d\d\s\d\d:\d\d:\d\d)\s.*$
^.*(\d\d\d\d[/.-]\d\d[/.-]\d\d\s\d\d:\d\d:\d\d)\s.*$

```

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/9/9eff9f75a6b94a67653f6c3a9770b2f9e0efdf15.png)

Started sending logs again from a remote Nginx server to Graylog and as you can see it created the field **new\_timestamp** and added another field called _ **gl2\_processing\_error** Invalid format: “2021/10/22 22:53:31” is malformed at “/10/22 22:53:31”._ I believe that’s what you were stating, correct?

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/3/3cdb476fb8e4d5abe837a3757b4b66cc40e776c2.png)

My next test was to send the Nginx log from the input to a stream and use the pipeline above but unfortunately that also did not work correctly. My pipeline knowledge is not that good to convert the Date/time ` yyyy/mm/dd` into something of `yyy-mm-dd`.

What you want could be done through a pipeline. It should be able to convert Date/Time and create a new field for you but I just don’t know how yet. There are other community members here that are very knowledgeable with this type of issue. I’ll keep trying and see what I can come up with unless someone here jumps in.

---

<div class="post-metadata">

**Author:** ![aft2d](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/aft2d/32/8547_2.png) [@aft2d](https://community.graylog.org/u/aft2d)\
**Post date:** [October 25, 2021, 1:07pm UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/9 "2021-10-25T13:07:59Z")

</div>

> [@Winklerschulbedarf](#):
>
> Ok so when trying to copy the needed timestamp from the message and convert it into another field i.e. timestamp2 everything works just fine.
> 
> ![grafik](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/7/711eb991e622a5716060add31a88d9f3c9c0cb97.png)
> 
> You can notice the difference of the format between the two timestamps.  
> Maybe this matters?

I can reproduce this. If you choose any name out of `timestamp` it works.  
If you use a flex date converter instead you even have the same format as in the default `timestamp` field. But the issue persists.

A guess:  
Maybe it’s related to message processing order. Like: Extractor → timestamp parser → converter  
This would result that at the step where graylog tries to parse the timestamp, the timestamp would be in the `2021/10/22 08:42:47` format and unable to read it?

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [October 26, 2021, 1:38am UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/10 "2021-10-26T01:38:31Z")

</div>

I found another post with the same issue. Might want to check it out.

> [@Nginx log access time stamps cannot be converted](https://community.graylog.org/t/nginx-log-access-time-stamps-cannot-be-converted/5325/6):
>
> See [http://docs.graylog.org/en/2.4/pages/pipelines/functions.html#parse-date](http://docs.graylog.org/en/2.4/pages/pipelines/functions.html#parse-date).

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [October 28, 2021, 4:46am UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/11 "2021-10-28T04:46:11Z")

</div>

Not sure if this helps, but I came across this post earlier today with a similar issue with Nginx

> [@Graylog exctractor specify timezone other than UTC](https://community.graylog.org/t/graylog-exctractor-specify-timezone-other-than-utc/3663):
>
> Hello team! I have a message with timestamp without specified timezone in it, but timezone actually -6 CST. It is not UTC. Message: “2017/12/30 15:19:57 [error] 27739#0: \*3722503 limiting requests, excess: 0.062 by zone “wp-login”, client: 10.189.252.6, server: [blog.com](http://blog.com), request: “POST /ame/wp-login.php HTTP/1.1”, host: “[www.ame.com](http://www.ame.com)”” With following GROK pattern: %{DATESTAMP\_EVENTLOG:timestamp;date;yyyy/MM/dd HH:mm:ss} Is there a way to specify at time parsing tell that timestamp in messag…

---

<div class="post-metadata">

**Author:** ![Winklerschulbedarf](https://avatars.discourse-cdn.com/v4/letter/w/ec9cab/32.png) [@Winklerschulbedarf](https://community.graylog.org/u/Winklerschulbedarf)\
**Post date:** [October 28, 2021, 8:46am UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/12 "2021-10-28T08:46:35Z")

</div>

Hello guys!  
Sorry i was very busy the last days and had no time to test out stuff.  
But i finally got it working again using a mix of extractor and pipeline.

Heres how i did it:

1. Create an extractor to copy the timestamp from the message into a second timestamp field.

 ![grafik](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/5/5b1ca1c10e01256ac42831a70465b3ae27a6e74c.png)

1. Create a pipeline on the stream that gets the messages you want the timestamps to get changed.

![grafik](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/f/f4f52f650b32ae223ae579b581a7e2a949f2c638.png)

Edit: You can also use **has\_field(“msg\_timestamp”)** instead of **true** if you want to.

1. Change the message processor order under System \> Configurations to the following:

 ![grafik](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/c/c509c5f7a8b4fbe6a1b2dfaedf3b274185c9828a.png)

Its important that the Message Filter Chain comes before the Pipeline Processor!

Of course you could do everything with the pipeline instead of using a mix but at this point i was just to lazy. I hope i didnt forget anything. If so just ask me and i will try to help you out.  
Thanks for the help guys!

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [October 28, 2021, 9:56pm UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/13 "2021-10-28T21:56:18Z")

</div>

Good Job 👍

Can you show the results?

---

<div class="post-metadata">

**Author:** ![Winklerschulbedarf](https://avatars.discourse-cdn.com/v4/letter/w/ec9cab/32.png) [@Winklerschulbedarf](https://community.graylog.org/u/Winklerschulbedarf)\
**Post date:** [October 29, 2021, 6:59am UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/14 "2021-10-29T06:59:49Z")

</div>

Hello there!  
Here are the requested results:

So what im doing is send a syslog message with powershell to the graylog server:

 ![grafik](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/b/bd088645c7ee998abbe70595f6854bf8d33f1d00.png)

In this case the message gets routed into a test stream. You can see that the timestamp has been updated:

 ![grafik](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/7/78819fd9b3e8974b97b42583a545733408e5c0b7.png)

Also i changed the pipeline to delete the “msg\_timestamp” field after the right timestamp has been updated:  
 ![grafik](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/4/4148bf23306cd9668369558d1cfeffb53b5a4932.png)

If you need something more just tell me!

---

<div class="post-metadata">

**Author:** ![aft2d](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/aft2d/32/8547_2.png) [@aft2d](https://community.graylog.org/u/aft2d)\
**Post date:** [October 29, 2021, 10:38am UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/15 "2021-10-29T10:38:37Z")

</div>

Great to hear 👍

But is this the intended way to do this?  
In my opinion, this is a work-around, but not a solution for the actual issue?

---

<div class="post-metadata">

**Author:** ![Winklerschulbedarf](https://avatars.discourse-cdn.com/v4/letter/w/ec9cab/32.png) [@Winklerschulbedarf](https://community.graylog.org/u/Winklerschulbedarf)\
**Post date:** [October 29, 2021, 11:58am UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/16 "2021-10-29T11:58:01Z")

</div>

Well as i said this problem occurred after the update to version 4.2.0.  
Before the update i only used the extractor to update the timestamp.  
Im not really sure if there is an intended way to do this.  
For me this is just a workaround and im still hoping that with the next patch i can use the extractor to update the timestamps again.

---

<div class="post-metadata">

**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\
**Post date:** [October 29, 2021, 11:48pm UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/17 "2021-10-29T23:48:52Z")

</div>

Hello

> [@aft2d](#):
>
> `2021/10/22 08:42:47` format and unable to read it

I believe you are correct. Do to Nginx _error.log_ file Date/Time you have to either convert the message before sending it or use a pipeline.  
Not sure why you would need to covert the date/time from the message field I noticed that in my lab it is already correct. Here is my Nginx error log message that is in the input. No extractors or pipelines.

 ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/2X/d/d918d52e563908a5c7ab7b09879eba8ad333557e.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\
**Post date:** [November 12, 2021, 11:48pm UTC](https://community.graylog.org/t/issues-with-date-type-converter-format-string/21524/18 "2021-11-12T23:48:59Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.