Failed to parse field with format date_time

[nix-power (Dima Zyuryaev)]

Env:
Graylog: 3.1.4 running in Docker, Ubuntu 18
Elasticsearch: 6.8.22 running on separate nodes

server.log has multiple errors that look the same:

2022-01-10T13:36:30.663Z ERROR [EventProcessorExecutionJob] Event processor <aggregation-v1/6109064c3437f400135be4bf> failed to execute: Unable to perform search query

failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [strict_date_optional_time||epoch_millis]failed to parse date field [2021-10-06 14:21:39.517] with format [strict_date_optional_time||epoch_millis]. (retry in 5000 ms)
org.graylog.events.processor.EventProcessorException: Unable to perform search query

failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [strict_date_optional_time||epoch_millis]failed to parse date field [2021-10-06 14:21:39.517] with format [strict_date_optional_time||epoch_millis].
	at org.graylog.events.processor.aggregation.PivotAggregationSearch.doSearch(PivotAggregationSearch.java:132) ~[graylog.jar:?]
	at org.graylog.events.processor.aggregation.AggregationEventProcessor.aggregatedSearch(AggregationEventProcessor.java:205) ~[graylog.jar:?]
	at org.graylog.events.processor.aggregation.AggregationEventProcessor.createEvents(AggregationEventProcessor.java:119) ~[graylog.jar:?]
	at org.graylog.events.processor.EventProcessorEngine.execute(EventProcessorEngine.java:92) ~[graylog.jar:?]
	at org.graylog.events.processor.EventProcessorExecutionJob.execute(EventProcessorExecutionJob.java:111) ~[graylog.jar:?]
	at org.graylog.scheduler.JobExecutionEngine.executeJob(JobExecutionEngine.java:166) ~[graylog.jar:?]
	at org.graylog.scheduler.JobExecutionEngine.lambda$handleTrigger$2(JobExecutionEngine.java:144) ~[graylog.jar:?]
	at com.codahale.metrics.Timer.time(Timer.java:137) ~[graylog.jar:?]
	at org.graylog.scheduler.JobExecutionEngine.handleTrigger(JobExecutionEngine.java:144) ~[graylog.jar:?]
	at org.graylog.scheduler.JobExecutionEngine.lambda$execute$0(JobExecutionEngine.java:119) ~[graylog.jar:?]
	at org.graylog.scheduler.worker.JobWorkerPool.lambda$execute$0(JobWorkerPool.java:110) ~[graylog.jar:?]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
	at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]

[gsmith]

Hello,

ailed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [
2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517]
 with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] 
 with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] 
 with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] 
 with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] 
 with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] with format [date_time]failed to parse date field [2021-10-06 14:21:39.517] 
 with format [strict_date_optional_time||epoch_millis]failed to parse date field [2021-10-06 14:21:39.517] with format [strict_date_optional_time||epoch_millis].

To be honest this could be a couple different issues. But the moral of the story is you have a field that can not be parsed because the Date/TIme is not correct format.

Do you have a custom index template?
Did you create a extractor on a INPUT?

With the lack of information given that is about the best I could do for you.
For a better understanding please look here.

How to Post a Question in the Community that Gets Responses

This platform is made with love for community discussions on the open source tool Graylog, it components and usage. Here’s a Graylog support-inspired template (thank you, @aaronsachs ) that’ll get responses: Description of your problem <!-- Use this section to describe the problem that you're encountering. Please include any screenshots or recordings of the problem you're running into.--> Description of steps you’ve taken to attempt to solve the issue <!-- Use this section to provide detail…

[nix-power (Dima Zyuryaev)]

I do have custom templates, but non of them related to mapping of time formats (using them only for elasticsearch ilm )
I will check now extractors on all my inputs to find something that may be related to changing date fromat.

[nix-power (Dima Zyuryaev)]

There are no extractors on inputs that somehow related to dealing with date fields.
Also i noticed that all these errors come from event processor

2022-01-11T06:31:18.355Z ERROR [EventProcessorExecutionJob] Event processor <aggregation-v1/60f955996cca640012242277> failed to execute: Unable to perform search query

[gsmith]

Hello,
Need to ask a couple questions.

• Was these error/s always shown? or did they just happen?
• If they are recent what was done to your environment prior to these errors?
• What have you tried to resolve this issue?