We’ve tried using "%{DATA:field}" which again works in the test tools and also works in the stream with the super low volume SYSTEM logs (one message per minute), but if we apply it to the pattern for TRAFFIC logs which are a much higher volume (10,000 messages per minute) the seizure happens.
It appears that both "%{DATA:field}" and %{QUOTEDSTRING:field} result in the ‘backtracking’ method that Drew mentioned earlier, which is too CPU intensive at scale. We’re hoping to find a pattern that works to parse the quoted string without being so computationally expensive.
We also just tried using "%{GREEDYDATA:characteristic_of_app}", on the TRAFFIC logs but Graylog seized on that as well.
We’ve created a new grok pattern called %{QUOTEDSTR} and then put %{QUOTEDSTR:characteristic_of_app}, into the TRAFFIC pattern. We tried setting %{QUOTEDSTR} to "([^"]*)", basically telling it “find anything that’s not a quote inbetween two quotes”, but no love - it seized, even though it worked via the test button.
We also noticed that in the TRAFFIC logs, sometimes the quoted string for characteristic_of_app does not appear in the message at all - some of the TRAFFIC log entries don’t have it, with nothing in-between the comma separators. So we tried using (?:"%{DATA:characteristic_of_app}")?, or (?:%QUOTEDSTRING:characteristic_of_app})?, in the TRAFFIC grok pattern to include a non-capturing group. This worked via the Test with Sample Data button with both a quoted string sample and a non quoted string sample, but in practice after saving the pattern it still caused Graylog to sieze.
We also noticed when using these new patterns that Graylog had actually successfully groked the characteristic_of_app field on some messages, namely those that occurred in the very short period of time (about 3 seconds) between saving the grok pattern and the seizure taking full effect. In these logs, the characteristic_of_app field exists and contains the exact contents of the quoted string.
Is it perhaps seizing when encountering a message without the quoted string, where characteristic_of_app would be NULL?
