Error while parsing

docgyan · June 7, 2021, 6:53am

Hello,

I have created a demonset of filebeat on azure kubernetes but seems like there is a parsing error which am not able to figure out. Kindly help me out.

I got the code from beats/deploy/kubernetes/filebeat at master · elastic/beats · GitHub

Below is the error:

2021-06-07T06:47:18.903431005Z2021-06-07T06:47:18.903Z ERROR [reader_docker_json] readjson/docker_json.go:204 Parse line error: parsing CRI timestamp: parsing time "2021/06/07" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "/06/07" as "-"

2021-06-07T06:47:18.903462505Z2021-06-07T06:47:18.903Z ERROR [reader_docker_json] readjson/docker_json.go:204 Parse line error: parsing CRI timestamp: parsing time "2021/06/07" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "/06/07" as "-"

2021-06-07T06:47:18.903469505Z2021-06-07T06:47:18.903Z ERROR [reader_docker_json] readjson/docker_json.go:204 Parse line error: parsing CRI timestamp: parsing time "2021/06/07" as "2006-01-02T15:04:05.999999999Z07:00": cannot parse "/06/07" as "-"

gsmith · June 8, 2021, 12:27am

Hello,
I’m not to familiar with

“demonset of filebeat on azure kubernetes”

but I did a google search for Parse line error: parsing CRI timestamp and came across these post. Maybe they can help you.

github.com/elastic/beats

Filebeat 6.4.0 chokes on docker continuation lines, "parsing CRI timestamp" errors

opened 01:58PM - 03 Sep 18 UTC

closed 02:27PM - 03 Sep 18 UTC

towolf

We upgraded to Filebeat 6.4.0 running in Kubernetes with regular Docker engine w…ith `json-file` logging. Now it begun to choke on some messages, trying to parse dates for CRI format. We do not have CRI format. The actual file on disk looks like this: ``` {"log":"{ \"timestamp\":\"2018-09-03 11:25:31.950\", \"level\":\"debug\", \"http_x {"log":"{ \"timestamp\":\"2018-09-03 11:25:31.951\", \"level\":\"debug\", \"http_x {"log":"{ \"timestamp\":\"2018-09-03 11:25:31.951\", \"level\":\"info\", \"http_x_ {"log":"{ \"timestamp\":\"2018-09-03 11:25:31.951\", \"level\":\"debug\", \"http_x {"log":"{ \"timestamp\":\"2018-09-03 11:25:32.361\", \"level\":\"debug\", \"http_x {"log":"ncies\\\" : [ {\\n \\\"currency_id\\\" : \\\"sample-client-current\ {"log":"\"user\\\" : \\\"200060\\\",\\n \\\"lang\\\" : \\\"de\\\"\\n }\\n {"log":"d\\\" : \\\"242219ea-c250-45a0-9070-d6fa65501eba\\\",\\n \\\"account_nu {"log":" \\\"19a8562e-78ae-45a5-842a-8a0e36206461\\\",\\n {"log":"{ \"timestamp\":\"2018-09-03 11:25:32.363\", \"level\":\"debug\", \"http_x {"log":"{ \"timestamp\":\"2018-09-03 11:25:32.364\", \"level\":\"info\", \"http_x_ {"log":"{ \"timestamp\":\"2018-09-03 11:25:38.023\", \"level\":\"info\", \"http_x_ {"log":"{ \"timestamp\":\"2018-09-03 11:25:38.023\", \"level\":\"info\", \"http_x_ {"log":"{ \"timestamp\":\"2018-09-03 11:25:38.023\", \"level\":\"debug\", \"http_x {"log":"{ \"timestamp\":\"2018-09-03 11:25:38.241\", \"level\":\"debug\", \"http_x {"log":"{ \"timestamp\":\"2018-09-03 11:25:38.241\", \"level\":\"debug\", \"http_x {"log":"{ \"timestamp\":\"2018-09-03 11:25:38.242\", \"level\":\"info\", \"http_x_ {"log":"{ \"timestamp\":\"2018-09-03 11:25:38.242\", \"level\":\"debug\", \"http_x {"log":"{ \"timestamp\":\"2018-09-03 11:25:38.792\", \"level\":\"debug\", \"http_x {"log":"encies\\\" : [ {\\n \\\"currency_id\\\" : \\\"sample-client-current {"log":"\\\"user\\\" : \\\"200060\\\",\\n \\\"lang\\\" : \\\"de\\\"\\n }\\ {"log":"id\\\" : \\\"242219ea-c250-45a0-9070-d6fa65501eba\\\",\\n \\\"account_n {"log":": \\\"19a8562e-78ae-45a5-842a-8a0e36206461\\\",\\n {"log":"{ \"timestamp\":\"2018-09-03 11:25:38.794\", \"level\":\"debug\", \"http_x {"log":"{ \"timestamp\":\"2018-09-03 11:25:38.796\", \"level\":\"info\", \"http_x_ ``` The errors printed are these: ``` 2018-09-03T11:25:17.962Z ERROR log/harvester.go:278 Read line error: parsing CRI timestamp: parsing time ""-101\\\",\\n" as "2006-01-02T15:04:05Z07:00": cannot parse ""-101\\\",\\n" as "2006"; File: /var/lib/docker/containers/2e9a3a7e4245431ab5c09cd794fed0f6c2e564ef9daf7aa1f8ddcebd96616577/2e9a3a7e4245431ab5c09cd794fed0f6c2e564ef9daf7aa1f8ddcebd96616577-json.log 2018-09-03T11:44:28.000Z ERROR log/harvester.go:278 Read line error: parsing CRI timestamp: parsing time "\\\",\\n" as "2006-01-02T15:04:05Z07:00": cannot parse "\\\",\\n" as "2006"; File: /var/lib/docker/containers/2e9a3a7e4245431ab5c09cd794fed0f6c2e564ef9daf7aa1f8ddcebd96616577/2e9a3a7e4245431ab5c09cd794fed0f6c2e564ef9daf7aa1f8ddcebd96616577-json.log 2018-09-03T11:44:38.001Z ERROR log/harvester.go:278 Read line error: parsing CRI timestamp: parsing time "\\\",\\n" as "2006-01-02T15:04:05Z07:00": cannot parse "\\\",\\n" as "2006"; File: /var/lib/docker/containers/2e9a3a7e4245431ab5c09cd794fed0f6c2e564ef9daf7aa1f8ddcebd96616577/2e9a3a7e4245431ab5c09cd794fed0f6c2e564ef9daf7aa1f8ddcebd96616577-json.log 2018-09-03T11:44:48.001Z ERROR log/harvester.go:278 Read line error: parsing CRI timestamp: parsing time "\\\",\\n" as "2006-01-02T15:04:05Z07:00": cannot parse "\\\",\\n" as "2006"; File: /var/lib/docker/containers/2e9a3a7e4245431ab5c09cd794fed0f6c2e564ef9daf7aa1f8ddcebd96616577/2e9a3a7e4245431ab5c09cd794fed0f6c2e564ef9daf7aa1f8ddcebd96616577-json.log ``` And so on and so forth.

github.com/elastic/beats

Invalid CRI error on Filebeat with docker log join enabled

opened 07:40AM - 31 Aug 18 UTC

closed 01:22PM - 31 Aug 18 UTC

wflyer

bug Filebeat containers v6.4.0

- Version: 6.4.0 - Operating System: Ubuntu 16.04 LTS - Steps to Reproduce: … On filebeat 6.4.0, with docker combine_partials enabled (which is default), sometimes docker log suddenly can't be processed such as below: ``` 2018-08-31T16:29:27.810+0900 ERROR log/harvester.go:278 Read line error: invalid CRI log; File: /var/lib/docker/containers/c37ca2c20d3a79d5430f2c967b104d6995f425ceb387394e19cc74218a505a1e/c37ca2c20d3a79d5430f2c967b104d6995f425ceb387394e19cc74218a505a1e-json.log ``` What I found until now is, the registry of the docker log file is incorrect, which is smaller than current file offset even when all log messages were processed. (the docker container has some logs larger than 16k) So, the docker containers that actively printing logs don't experience this issue. But when some logs closed due to infrequent access (larger than `close_inactive`), they experience this kind of issue when it re-opened. Also, I could reproduce this error even when I restart filebeat (eg.`service filebeat restart`) maybe because the harvester starts to read from the specified incorrect offset written in the registry file.

Hope that helps.

system · June 22, 2021, 12:28am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Log order - docker, filebeat Graylog Central (peer support) pipeline-rules , sidecar , filebeat-linux , nosendlogfblx	4	887	September 18, 2020
Error in graylog when timestamp field data appending timezone by sending via logstash Graylog Central (peer support)	5	621	July 25, 2019
Graylog timestamp issue for filebeat logs Graylog Central (peer support)	6	684	January 4, 2019
Timestamp from log line in nginx JSON logs via filebeat / beats Graylog Tech Challenges	4	3419	December 7, 2021
Tomcat timestamp parsing problem Graylog Central (peer support)	3	492	October 28, 2022

Error while parsing

Related Topics