1. Describe your incident:
After graylog upgrade to v4.2.6 we get these msgs:
gl2_processing_error
Replaced invalid timestamp value in message <2f44c541-98c8-11ec-88cc-1458d05629d8> with current time - Value <2022-02-28T18:56:50+00:00> caused exception: Invalid format: "2022-02-28T18:56:50+00:00" is malformed at "T18:56:50+00:00".
2. Describe your environment:
log ingest setup: (rsyslog and graylog and elastic are on the same host)
device_sends_logs > rsyslog > graylog_input > extractor > stream > elastic_index_X
- OS Information:
centos7
kernel 3.10.0-1160.11.1.el7.x86_64
- Package Version:
graylog-server-4.2.6-1.noarch and elasticsearch-7.17.0-1.x86_64
- Service logs, configurations, and environment variables:
For rsyslog we use this json template:
template(name="syslog_json" type="list" option.json="on") {
constant(value="{")
constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"source\":\"") property(name="hostname")
constant(value="\",\"ip_address\":\"") property(name="fromhost-ip")
constant(value="\",\"application_name\":\"") property(name="programname")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\",\"tag\":\"") property(name="syslogtag")
constant(value="\",\"level\":\"") property(name="syslogseverity")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\"}")
}
# HP
if $syslogfacility-text == 'local3' then @[::1]:5143;syslog_json
3. What steps have you already taken to try and solve the problem?
Looked around here:
As a test I removed all the extractors and my msgs came in as so and with no gl2_processing_error
{"timestamp":"2022-02-28T18:56:49+00:00","source":"<redacted>.ip6.arpa","ip_address":"<redacted>","application_name":"<redacted>","message":" 00435 ports: port B1 is Blocked by AAA","tag":"<redacted>","level":"6","facility":"local3"}
Once I add the extractors back, I get the gl2_processing_error msg.
My extractors:
{
"extractors": [
{
"title": "json_extractor",
"extractor_type": "json",
"converters": [],
"order": 0,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "",
"extractor_config": {
"flatten": true,
"list_separator": ", ",
"kv_separator": "=",
"key_prefix": "",
"key_separator": "_",
"replace_key_whitespace": true,
"key_whitespace_replacement": "_"
},
"condition_type": "none",
"condition_value": ""
},
{
"title": "hostname_from_fqdn; bug with FW: 16_02_0021",
"extractor_type": "regex_replace",
"converters": [],
"order": 0,
"cursor_strategy": "copy",
"source_field": "source",
"target_field": "source",
"extractor_config": {
"regex": "(.*)\\.ip[4,6].arpa",
"replacement": "$1",
"replace_all": true
},
"condition_type": "regex",
"condition_value": "(.*)\\.ip[4,6].arpa"
},
{
"title": "application_name_extractor_1",
"extractor_type": "regex",
"converters": [],
"order": 0,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "application_name",
"extractor_config": {
"regex_value": "^([\\w\\-]+):\\s\\s"
},
"condition_type": "none",
"condition_value": ""
},
{
"title": "application_name_extractor_2",
"extractor_type": "regex",
"converters": [],
"order": 0,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "application_name",
"extractor_config": {
"regex_value": "^\\d+\\s([\\w\\-]+):\\s\\s"
},
"condition_type": "none",
"condition_value": ""
}
],
"version": "4.2.6"
}
On this comment processing error after migrating to 4.2.0-3 · Issue #11495 · Graylog2/graylog2-server · GitHub it says to add some temp_timestamp extractor > json_extractor > then apply the converter…
I attempted this as so:
{
"extractors": [
{
"title": "timestamp_extractor",
"extractor_type": "regex_replace",
"converters": [],
"order": 0,
"cursor_strategy": "copy",
"source_field": "message",
"target_field": "temp_timestamp",
"extractor_config": {
"regex": "^\\{\\\"timestamp\\\":\\\"(\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\+\\d{2}:\\d{2})\\\",.*",
"replacement": "$1",
"replace_all": false
},
"condition_type": "regex",
"condition_value": "^\\{\\\"timestamp\\\":\\\"(\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\+\\d{2}:\\d{2})\\\",.*"
},
{
"title": "json_extractor",
...
"title": "hostname_from_fqdn; bug with FW: 16_02_0021",
...
"title": "application_name_extractor_1",
...
"title": "application_name_extractor_2",
...
{
"title": "timestamp_replace",
"extractor_type": "regex_replace",
"converters": [
{
"type": "date",
"config": {
"time_zone": "Etc/UTC",
"locale": "und",
"date_format": "yyyy-MM-dd HH:mm:ss,SSS"
}
}
],
"order": 0,
"cursor_strategy": "copy",
"source_field": "temp_timestamp",
"target_field": "timestamp",
"extractor_config": {
"regex": "(\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\+\\d{2}:\\d{2})",
"replacement": "$1",
"replace_all": true
},
"condition_type": "regex",
"condition_value": "(\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\+\\d{2}:\\d{2})"
}
],
"version": "4.2.6"
}
But got this error msg
gl2_processing_error
Could not apply converter [DATE] of extractor <timestamp_replace (5a618c30-98ce-11ec-88cc-1458d05629d8)> - Invalid format: "2022-02-28T19:53:59+00:00" is malformed at "T19:53:59+00:00"., Replaced invalid timestamp value in message <2bb9d023-98d0-11ec-88cc-1458d05629d8> with current time - Value <2022-02-28T19:53:59+00:00> caused exception: Invalid format: "2022-02-28T19:53:59+00:00" is malformed at "T19:53:59+00:00"
my fields looks like so when they are stored in elastic:
temp_timestamp
2022-02-28 19:53:59.000 +00:00
timestamp
2022-02-28 19:53:59.587 +00:00
4. How can the community help?
I would have thought that the 1/5 patch would have fixed this Perform message timestamp conversion after processing (#11884) (#11915) · Graylog2/graylog2-server@80a07bd · GitHub
What am I doing wrong?
Was the patch supposed to address this?
Any input is much appreciated!
Thank you,
Dave
