# Use of ${QUOTEDSTRING} causes Graylog to seize - Part II **URL:** \ **Category:** Graylog Central (peer support)\ **Tags:** grok-patternspl\ **Created:** [December 30, 2024, 6:18pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564 "2024-12-30T18:18:52Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![marziglt](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@marziglt](https://community.graylog.org/u/marziglt)\ **Post date:** [December 30, 2024, 6:18pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/1 "2024-12-30T18:18:52Z")
**1. Describe your incident:** We are creating a grok pattern to parse a complex message that contains quoted strings with commas inside the quotes. Whenever we use the default ${QUOTEDSTRING} or ${QS} grok patterns against these logs, Graylog ceases the indexing of all logs to OpenSearch until the Graylog server is rebooted. The problem does not occur when the use of ${QUOTEDSTRING} or ${QS} is removed from the grok pattern. **2. Describe your environment:** We’re running Graylog 6.1 and OpenSearch 2.15 in a two-server configuration on a security hardened Oracle Linux 8 OS. **3. What steps have you already taken to try and solve the problem?** - We found nothing in the Graylog server logs or OpenSearch cluster logs related to the issue. - We created a our own pattern called ${QUOTEDSTR} with the regex “(.\*?” but using this pattern also caused Graylog to seize with the exact same symptoms. **4. How can the community help?** - How can we grok quoted strings without using ${QUOTEDSTRING} or ${QS}? - How can we use ${QUOTEDSTRING} or ${QS} without Graylog seizing? ---
**Author:** ![drewmiranda-gl](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/drewmiranda-gl/32/13376_2.png) [@drewmiranda-gl](https://community.graylog.org/u/drewmiranda-gl)\ **Post date:** [December 31, 2024, 7:15pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/2 "2024-12-31T19:15:17Z")
Howdy! Are you able to provide both your grok patten and a couple of sample texts that the grok pattern is meant to be applied to? I ask because GROK (for the purposes of this reply i’ll be calling this regular expressions) has a very large degree of variability with how the patterns are constructed and the text it parses can greatly affect how it performs. Use off `.*?` can be very expensive (CPU wise) due to [backtracking](https://stackoverflow.com/questions/9011592/in-regular-expressions-what-is-a-backtracking-back-referencing) which is to say the regular expression parser goes to the END of the string and tests each character one by one. If the text is long, this can be VERY CPU intensive. This can be demonstrated using a regular expression debugger, like the one provided via [https://regex101.com/](https://regex101.com/) that shows the number of steps required to have the regular expression parse the text. Lastly, can you further quantify or qualify what “causes Graylog to seize” means? In technical terms, does this mean the ALL CPUs available to graylog are at 100% utilization and graylog stops ingesting messages? I would also expect the process buffer to become 100% filled in this scenario. Depending on message throughput (both messages per second AND total size of messages per second) the Graylog server(s) may need additional CPU but i’d like to debug the regex pattern to see if there is an easy fix there. Thanks! ---
**Author:** ![marziglt](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@marziglt](https://community.graylog.org/u/marziglt)\ **Post date:** [December 31, 2024, 10:08pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/3 "2024-12-31T22:08:12Z")
Sure - here is a sanitized version: Pattern: > %{WORD:logsrc}.div.company.com %{NUMBER:num},%{DATA:receive\_time},%{NUMBER:serial},%{WORD:type},%{WORD:subtype},%{NUMBER:port},%{DATA:time\_generated},%{DATA:src},%{DATA:dst},%{DATA:natsrc},%{DATA:natdst},%{DATA:rule},%{DATA:srcuser},%{DATA:dstuser},%{DATA:app},%{DATA:vsys},%{DATA:from},%{DATA:to},%{DATA:inbound\_if},%{DATA:outboundif},%{DATA:logset},%{DATA:unknown\_time},%{NUMBER:sessionid},%{NUMBER:repeatcnt},%{NUMBER:sport},%{NUMBER:dport},%{NUMBER:natsport},%{NUMBER:natdport},%{DATA:flags},%{DATA:proto},%{DATA:action},%{NUMBER:bytes},%{NUMBER:bytes\_sent},%{NUMBER:bytes\_received},%{NUMBER:packets},%{DATA:start},%{NUMBER:elapsed},%{DATA:category},%{DATA:unknown\_field1},%{DATA:seqno},%{DATA:actionflags},%{DATA:srcloc},%{DATA:dstloc},%{DATA:unknown\_field2},%{DATA:pkts\_sent},%{DATA:pkts\_received},%{DATA:session\_end\_reason},%{DATA:dg\_hier\_level\_1},%{DATA:dg\_hier\_level\_2},%{DATA:dg\_hier\_level\_3},%{DATA:dg\_hier\_level\_4},%{DATA:vsys\_name},%{DATA:device\_name},%{DATA:action\_source},%{DATA:src\_uuid},%{DATA:dst\_uuid},%{DATA:tunnelid},%{DATA:monitortag},%{DATA:parent\_session\_id},%{DATA:parent\_start\_time},%{DATA:tunnel},%{NUMBER:assoc\_id},%{NUMBER:chunks},%{NUMBER:chunks\_sent},%{NUMBER:chunks\_received},%{DATA:rule\_uuid},%{DATA:http2\_connection},%{DATA:link\_change\_count},%{DATA:policy\_id},%{DATA:link\_switches},%{DATA:sdwan\_cluster},%{DATA:sdwan\_device\_type},%{DATA:sdwan\_cluster\_type},%{DATA:sdwan\_site},%{DATA:dynusergroup\_name},%{DATA:xff\_ip},%{DATA:src\_category},%{DATA:src\_profile},%{DATA:src\_model},%{DATA:src\_vendor},%{DATA:src\_osfamily},%{DATA:src\_osversion},%{DATA:src\_host},%{DATA:src\_mac},%{DATA:dst\_category},%{DATA:dst\_profile},%{DATA:dst\_model},%{DATA:dst\_vendor},%{DATA:dst\_osfamily},%{DATA:dst\_osversion},%{DATA:dst\_host},%{DATA:dst\_mac},%{DATA:container\_id},%{DATA:pod\_namespace},%{DATA:pod\_name},%{DATA:src\_edl},%{DATA:dst\_edl},%{DATA:hostid},%{DATA:serialnumber},%{DATA:src\_dag},%{DATA:dst\_dag},%{DATA:session\_owner},%{DATA:high\_res\_timestamp},%{DATA:nssai\_sst},%{DATA:nssai\_sd},%{DATA:subcategory\_of\_app},%{DATA:category\_of\_app},%{DATA:technology\_of\_app},%{DATA:risk\_of\_app},%{QUOTEDSTRING:characteristic\_of\_app},%{DATA:container\_of\_app},%{DATA:tunneled\_app},%{DATA:is\_saas\_of\_app},%{DATA:sanctioned\_state\_of\_app},%{DATA:offloaded},%{DATA:traffic\_type}, Message: > [panorama.div.company.com](http://panorama.div.company.com) 1,2024/12/02 15:18:02,024101003988,TRAFFIC,end,2817,2024/12/02 15:18:02,10.101.19.7,10.101.159.130,0.0.0.0,0.0.0.0,inside-in\_46,msrpc-base,vdiv1,PROD.Internal,PROD.DIVNET,ethernet1/2,ethernet1/1,default,2024/12/02 15:18:02,655765,1,61636,49681,0,0,0x401a,tcp,allow,4896,3798,1098,16,2024/12/02 15:17:19,29,any,7432514093394733498,0x8000000000000000,10.0.0.0-10.255.255.255,10.0.0.0-10.255.255.255,9,7,tcp-rst-from-client,20,11,0,0,PA2.NJ17,from-policy,0,0,N/A,0,0,0,0,1d654101-d3e3-774e-df69-82f43a334666,0,0,2024-12-02T15:18:02.940-05:00,infrastructure,networking,network-protocol,2,“has-known-vulnerability,tunnel-other-application,pervasive-use”,msrpc,untunneled,no,no,0,NonProxyTraffic, It’s a Palo Alto Networks PANOS TRAFFIC log. The “characteristic\_of\_app” field is a long-ish quoted string that includes commas within the quotes. When we use the _Test with Sample Data_ button it groks the sample log successfully, but whenever we save the grok pattern with _%{QUOTEDSTRING:characteristic\_of\_app}_ defined, the output meter goes to zero and no further logs are indexed to OpenSearch, even though the input meter still runs as usual. We noticed using the _top_ command that CPU utilization is normally 20-30% but when “seized” it will jump to over 400%. If we remove _%{QUOTEDSTRING:characteristic\_of\_app}_ from the grok pattern and then either reboot or restart the _graylog-server_ service, the output to OpenSearch will resume as normal. ---
**Author:** ![patrickmann](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/patrickmann/32/9091_2.png) [@patrickmann](https://community.graylog.org/u/patrickmann)\ **Post date:** [January 1, 2025, 12:00pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/4 "2025-01-01T12:00:18Z")
When I looked at the first time you reported this I found that the grok pattern simply did not match the message. The mismatch occurred before the quoted string pattern was encountered. Have you verified that the match actually failed at quoted string? To narrow this down, can you reproduce this with a shorter test message and simpler grok pattern? ---
**Author:** ![drewmiranda-gl](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/drewmiranda-gl/32/13376_2.png) [@drewmiranda-gl](https://community.graylog.org/u/drewmiranda-gl)\ **Post date:** [January 2, 2025, 2:47pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/5 "2025-01-02T14:47:41Z")
Thank you @marziglt much appreciated. Can you share the context that this grok pattern is used? For example, via an extractor? Via a pipeline rule? Also to expand on what Patrick said, it appears the Grok pattern does not match the sample log message, specifically it matches up to `NUMBER:natsport` ```auto %{WORD:logsrc}.div.company.com %{NUMBER:num},%{DATA:receive_time},%{NUMBER:serial},%{WORD:type},%{WORD:subtype},%{NUMBER:port},%{DATA:time_generated},%{DATA:src},%{DATA:dst},%{DATA:natsrc},%{DATA:natdst},%{DATA:rule},%{DATA:srcuser},%{DATA:dstuser},%{DATA:app},%{DATA:vsys},%{DATA:from},%{DATA:to},%{DATA:inbound_if},%{DATA:outboundif}%{DATA:logset},%{DATA:unknown_time},%{NUMBER:sessionid},%{NUMBER:repeatcnt},%{NUMBER:sport},%{NUMBER:dport},%{NUMBER:natsport}, ``` which matches ```auto panorama.div.company.com 1,2024/12/02 15:18:02,024101003988,TRAFFIC,end,2817,2024/12/02 15:18:02,10.101.19.7,10.101.159.130,0.0.0.0,0.0.0.0,inside-in_46,msrpc-base,vdiv1,PROD.Internal,PROD.DIVNET,ethernet1/2,ethernet1/1,default,2024/12/02 15:18:02,655765,1,61636,49681,0,0, ``` The next part of the grok pattern `%{NUMBER:natdport},` fails to match the next part of the message which is `0x401a`. It seems the grok pattern is off by one? The text its trying ot match appears to be the next grok pattern: `%{DATA:flags}` which would correctly match `0x401a`. I’m not sure if this is the cause of your issue though. For what its worth, [Graylog Illuminate parses Palo Alto](https://go2docs.graylog.org/illuminate-current/content_packs/palo_alto_content_pack.html) logs out of the box. Thanks, Drew ---
**Author:** ![marziglt](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@marziglt](https://community.graylog.org/u/marziglt)\ **Post date:** [January 2, 2025, 3:16pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/6 "2025-01-02T15:16:40Z")
We don’t have a shorter message but we encounter the exact same issue with similar fields in the PANOS THREAT and SYSTEM logs (separate patterns/rules), with the same symptoms - if we add %{QUOTEDSTRING} to the grok pattern, all indexing ceases until it’s removed and restarted. ---
**Author:** ![marziglt](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@marziglt](https://community.graylog.org/u/marziglt)\ **Post date:** [January 2, 2025, 3:46pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/7 "2025-01-02T15:46:41Z")
It’s implemented via a pipeline rule. In a previous stage, we determine and set the type of log (TRAFFIC|THREAT|SYSTEM|CONFIG|HIPMATCH|CORRELATION|ALERT|APP|AUTH|SAML|USERID) as _palo\_type_ and then in the next stage we use multiple rules to key on _palo\_type_ and apply the appropriate grok pattern using _Extract grok to fields_ via the Rule Builder. There are similar quoted string fields in the THREAT and SYSTEM log types and they also produce the same symptoms whenever %{QUOTEDSTRING} is added to the grok pattern. As for the pattern not matching, that’s strange - in Graylog it works (and aligns) via the _Test with Sample Data_ button all the way through to _risk\_of\_app_ field (just before the quoted string field _characteristic\_of\_app_). We keep the pattern truncated at that point and have been processing the logstream that way for weeks now. ---
**Author:** ![marziglt](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@marziglt](https://community.graylog.org/u/marziglt)\ **Post date:** [January 2, 2025, 7:44pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/8 "2025-01-02T19:44:50Z")
Today we were able to successfully implement a grok pattern on the PANOS SYSTEM log type which also has a quoted string containing commas as a field called _opaque_. Instead of using _%{QUOTEDSTRING:opaque},_ we used _“%{DATA:opaque}”,_ which works fine with no issues. When we applied the same solution to the grok pattern of TRAFFIC logs using _“%{DATA:characteristic\_of\_app}”,_ the seize problem returned until that part of the pattern was removed and the graylog-server service restarted. We even tried _“%{DATA:char\_of\_app}”,_ in case the field name was too long, but same result. One notable difference between these two log types is that in a five minute time-frame, Graylog will receive via input around five SYSTEM log messages and over 50,000 TRAFFIC log messages. It’s able to handle anything we’ve thrown at it with ease, including the majority of the TRAFFIC grok pattern, but it seizes whenever we try to expand the pattern to grok the _characteristic\_of\_app_ field, regardless of method. ---
**Author:** ![marziglt](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@marziglt](https://community.graylog.org/u/marziglt)\ **Post date:** [January 3, 2025, 2:20pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/9 "2025-01-03T14:20:38Z")
Today we expanded the grok pattern for the PANOS THREAT log type, which includes a quoted string field with commas named _url\_category\_list_ which we attempted to grok as _“%{DATA:url\_category\_list}”,_ and which worked when using the _Test with Sample Data_ button. We normally input around 20,000 THREAT messages in a five-minute period. Shortly after saving the grok pattern, Graylog seized in the same fashion - using the _top_ command we noticed CPU utilization was around 600% until we removed _url\_category\_list_ from the grok pattern and restarted _graylog-server_. We also noticed after restarting _graylog-server_ that on the Graylog console page when the output meter begins moving again, it will initially show an abnormally large output (thousands of messages) for about five or six frames (frame = update every two seconds) before resuming a normal level of output (hundreds of messages). It appears to be emptying out a buffer that built up during the seizure. We left the THREAT grok pattern truncated just before the offending _url\_category\_list_ field and the server is now happily puttering along at about 25-40% CPU utilization. ---
**Author:** ![patrickmann](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/patrickmann/32/9091_2.png) [@patrickmann](https://community.graylog.org/u/patrickmann)\ **Post date:** [January 7, 2025, 2:25pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/10 "2025-01-07T14:25:04Z")
I don’t have a simple answer. The messages and patterns are complex; and I have no idea how your pipeline rules and stages are structured. Clearly grok-matching is failing. I can’t tell yet, whether there is a bug in the pipeline rule processor. I think you need to start with the basics and work up from there: - The sample pattern does not match the sample message string. There are multiple discrepancies. Please use something like [https://grokdebugger.com/](https://grokdebugger.com/) to ensure that your pattern matches the sample messages. - Make sure the string presented to grok matches your expectation, e.g. by outputting it with the debug function from within the rule, before calling grok. - Validate the pipeline by applying it to test messages that you manually submit before using it in the production system. ---
**Author:** ![patrickmann](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/patrickmann/32/9091_2.png) [@patrickmann](https://community.graylog.org/u/patrickmann)\ **Post date:** [January 7, 2025, 3:21pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/11 "2025-01-07T15:21:36Z")
Meanwhile I discovered a [related issue](https://github.com/Graylog2/graylog2-server/issues/19975): Grok processor may fail silently due to stack overflow when processing large messages. The log then contains something like this: > 2024-07-12 09:35:25,618 WARN : org.glassfish.jersey.server.ServerRuntime$Responder - An exception mapping did not successfully produce and processed a response. Logging the exception propagated to the default exception mapper. > java.lang.StackOverflowError: null GL obviously needs to do better at bubbling up this problem. But eliminating the limitation will be more tricky. If this is the root cause of your problem, you can mitigate it by splitting up grok extraction into several steps. Or switching to regex. ---
**Author:** ![marziglt](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@marziglt](https://community.graylog.org/u/marziglt)\ **Post date:** [January 7, 2025, 7:29pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/12 "2025-01-07T19:29:51Z")
Here is a sanitized version of what we’re currently using (up to _risk\_of\_app_) that we confirmed works via Grok Debugger: Pattern: > %{WORD:logsrc}.div.company.com %{NUMBER:num},%{DATA:receive\_time},%{NUMBER:serial},%{WORD:type},%{WORD:subtype},%{NUMBER:unknown\_num},%{DATA:time\_generated},%{DATA:src},%{DATA:dst},%{DATA:natsrc},%{DATA:natdst},%{DATA:rule},%{DATA:srcuser},%{DATA:dstuser},%{DATA:app},%{DATA:vsys},%{DATA:from},%{DATA:to},%{DATA:inbound\_if},%{DATA:outboundif},%{DATA:logset},%{DATA:unknown\_time},%{NUMBER:sessionid},%{NUMBER:repeatcnt},%{NUMBER:sport},%{NUMBER:dport},%{NUMBER:natsport},%{NUMBER:natdport},%{DATA:flags},%{DATA:proto},%{DATA:action},%{NUMBER:bytes},%{NUMBER:bytes\_sent},%{NUMBER:bytes\_received},%{NUMBER:packets},%{DATA:start},%{NUMBER:elapsed},%{DATA:category},%{DATA:unknown\_field1},%{DATA:seqno},%{DATA:actionflags},%{DATA:srcloc},%{DATA:dstloc},%{DATA:unknown\_field2},%{DATA:pkts\_sent},%{DATA:pkts\_received},%{DATA:session\_end\_reason},%{DATA:dg\_hier\_level\_1},%{DATA:dg\_hier\_level\_2},%{DATA:dg\_hier\_level\_3},%{DATA:dg\_hier\_level\_4},%{DATA:vsys\_name},%{DATA:device\_name},%{DATA:action\_source},%{DATA:src\_uuid},%{DATA:dst\_uuid},%{DATA:tunnelid},%{DATA:monitortag},%{DATA:parent\_session\_id},%{DATA:parent\_start\_time},%{DATA:tunnel},%{NUMBER:assoc\_id},%{NUMBER:chunks},%{NUMBER:chunks\_sent},%{NUMBER:chunks\_received},%{DATA:rule\_uuid},%{DATA:http2\_connection},%{DATA:link\_change\_count},%{DATA:policy\_id},%{DATA:link\_switches},%{DATA:sdwan\_cluster},%{DATA:sdwan\_device\_type},%{DATA:sdwan\_cluster\_type},%{DATA:sdwan\_site},%{DATA:dynusergroup\_name},%{DATA:xff\_ip},%{DATA:src\_category},%{DATA:src\_profile},%{DATA:src\_model},%{DATA:src\_vendor},%{DATA:src\_osfamily},%{DATA:src\_osversion},%{DATA:src\_host},%{DATA:src\_mac},%{DATA:dst\_category},%{DATA:dst\_profile},%{DATA:dst\_model},%{DATA:dst\_vendor},%{DATA:dst\_osfamily},%{DATA:dst\_osversion},%{DATA:dst\_host},%{DATA:dst\_mac},%{DATA:container\_id},%{DATA:pod\_namespace},%{DATA:pod\_name},%{DATA:src\_edl},%{DATA:dst\_edl},%{DATA:hostid},%{DATA:serialnumber},%{DATA:src\_dag},%{DATA:dst\_dag},%{DATA:session\_owner},%{DATA:high\_res\_timestamp},%{DATA:nssai\_sst},%{DATA:nssai\_sd},%{DATA:subcategory\_of\_app},%{DATA:category\_of\_app},%{DATA:technology\_of\_app},%{DATA:risk\_of\_app},“%{DATA:characteristic\_of\_app}”,%{DATA:container\_of\_app},%{DATA:tunneled\_app},%{DATA:is\_saas\_of\_app},%{DATA:sanctioned\_state\_of\_app},%{DATA:offloaded},%{DATA:traffic\_type}, Message: > [panorama.div.company.com](http://panorama.div.company.com) 1,2024/12/02 15:18:02,024680993766,TRAFFIC,end,2897,2024/12/02 15:18:02,10.88.151.8,10.99.101.17,0.0.0.0,0.0.0.0,inernal-in\_37,msrpc-base,vdiv1,PROD.Internal,PROD.DIVNET,ethernet1/2,ethernet1/1,default,2024/12/02 15:18:02,655765,1,61636,49681,0,0,0x401a,tcp,allow,4896,3798,1098,16,2024/12/02 15:17:19,29,any,7432514093394733498,0x8000000000000000,10.0.0.0-10.255.255.255,10.0.0.0-10.255.255.255,9,7,tcp-rst-from-client,20,11,0,0,PA2.FL17,from-policy,0,0,N/A,0,0,0,0,1c563515-f2e7-227d-dc61-76f72a114777,0,0,2024-12-02T15:18:02.940-05:00,infrastructure,networking,network-protocol,2,“has-known-vulnerability,tunnel-other-application,pervasive-use”,msrpc,untunneled,no,no,0,NonProxyTraffic, Same symptoms are happening: if we truncate the pattern after _%{DATA:risk\_of\_app},_ it works fine and groks the messages up to that point - but if we add _“%{DATA:characteristic\_of\_app}”,_ or _%{QUOTEDSTRING:characteristic\_of\_app},_ Graylog will seize. When the seizure happens there are no WARN or ERROR entries in the graylog-server log - the CPU usage spikes to around 400% and remains that high until the _characteristic\_of\_app_ grok is removed and the graylog-server service is restarted. ---
**Author:** ![patrickmann](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/patrickmann/32/9091_2.png) [@patrickmann](https://community.graylog.org/u/patrickmann)\ **Post date:** [January 8, 2025, 7:57am UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/13 "2025-01-08T07:57:56Z")
I’m puzzled, because when I paste this into grokdebugger it still starts failing at `natsport`. Please check your server log for the stack overflow exception I mentioned above (or other uncaught exceptions arising in pipeline processing). If that’s the main problem, then it’s not about the patterns as such, but the complexity of the data and pattern. ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/0/2/028a5f4a5bd6fd931c77f2abdd1f1d1f63c6bdc9.png) ---
**Author:** ![marziglt](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@marziglt](https://community.graylog.org/u/marziglt)\ **Post date:** [January 8, 2025, 4:34pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/14 "2025-01-08T16:34:32Z")
We searched the _graylog-server_ logs, but found no stack overflow exception or any entry related to the seizures. ---
**Author:** ![marziglt](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@marziglt](https://community.graylog.org/u/marziglt)\ **Post date:** [January 8, 2025, 5:56pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/15 "2025-01-08T17:56:31Z")
These were triply for sure checked and verified as working on [Grok Debugger](https://grokdebugger.com/). Pattern: %{WORD:logsrc}.div.company.com %{NUMBER:num},%{DATA:receive\_time},%{NUMBER:serial},%{WORD:type},%{WORD:subtype},%{NUMBER:unknown\_num},%{DATA:time\_generated},%{DATA:src},%{DATA:dst},%{DATA:natsrc},%{DATA:natdst},%{DATA:rule},%{DATA:srcuser},%{DATA:dstuser},%{DATA:app},%{DATA:vsys},%{DATA:from},%{DATA:to},%{DATA:inbound\_if},%{DATA:outboundif},%{DATA:logset},%{DATA:unknown\_time},%{NUMBER:sessionid},%{NUMBER:repeatcnt},%{NUMBER:sport},%{NUMBER:dport},%{NUMBER:natsport},%{NUMBER:natdport},%{DATA:flags},%{DATA:proto},%{DATA:action},%{NUMBER:bytes},%{NUMBER:bytes\_sent},%{NUMBER:bytes\_received},%{NUMBER:packets},%{DATA:start},%{NUMBER:elapsed},%{DATA:category},%{DATA:unknown\_field1},%{DATA:seqno},%{DATA:actionflags},%{DATA:srcloc},%{DATA:dstloc},%{DATA:unknown\_field2},%{DATA:pkts\_sent},%{DATA:pkts\_received},%{DATA:session\_end\_reason},%{DATA:dg\_hier\_level\_1},%{DATA:dg\_hier\_level\_2},%{DATA:dg\_hier\_level\_3},%{DATA:dg\_hier\_level\_4},%{DATA:vsys\_name},%{DATA:device\_name},%{DATA:action\_source},%{DATA:src\_uuid},%{DATA:dst\_uuid},%{DATA:tunnelid},%{DATA:monitortag},%{DATA:parent\_session\_id},%{DATA:parent\_start\_time},%{DATA:tunnel},%{NUMBER:assoc\_id},%{NUMBER:chunks},%{NUMBER:chunks\_sent},%{NUMBER:chunks\_received},%{DATA:rule\_uuid},%{DATA:http2\_connection},%{DATA:link\_change\_count},%{DATA:policy\_id},%{DATA:link\_switches},%{DATA:sdwan\_cluster},%{DATA:sdwan\_device\_type},%{DATA:sdwan\_cluster\_type},%{DATA:sdwan\_site},%{DATA:dynusergroup\_name},%{DATA:xff\_ip},%{DATA:src\_category},%{DATA:src\_profile},%{DATA:src\_model},%{DATA:src\_vendor},%{DATA:src\_osfamily},%{DATA:src\_osversion},%{DATA:src\_host},%{DATA:src\_mac},%{DATA:dst\_category},%{DATA:dst\_profile},%{DATA:dst\_model},%{DATA:dst\_vendor},%{DATA:dst\_osfamily},%{DATA:dst\_osversion},%{DATA:dst\_host},%{DATA:dst\_mac},%{DATA:container\_id},%{DATA:pod\_namespace},%{DATA:pod\_name},%{DATA:src\_edl},%{DATA:dst\_edl},%{DATA:hostid},%{DATA:serialnumber},%{DATA:src\_dag},%{DATA:dst\_dag},%{DATA:session\_owner},%{DATA:high\_res\_timestamp},%{DATA:nssai\_sst},%{DATA:nssai\_sd},%{DATA:subcategory\_of\_app},%{DATA:category\_of\_app},%{DATA:technology\_of\_app},%{DATA:risk\_of\_app},“%{DATA:characteristic\_of\_app}”,%{DATA:container\_of\_app},%{DATA:tunneled\_app},%{DATA:is\_saas\_of\_app},%{DATA:sanctioned\_state\_of\_app},%{DATA:offloaded},%{DATA:traffic\_type}, Sample: [panorama.div.company.com](http://panorama.div.company.com) 1,2024/12/02 15:18:02,024680993766,TRAFFIC,end,2897,2024/12/02 15:18:02,10.88.151.8,10.99.101.17,0.0.0.0,0.0.0.0,inernal-in\_37,msrpc-base,vdiv1,PROD.Internal,PROD.DIVNET,ethernet1/2,ethernet1/1,default,2024/12/02 15:18:02,655765,1,61636,49681,0,0,0x401a,tcp,allow,4896,3798,1098,16,2024/12/02 15:17:19,29,any,7432514093394733498,0x8000000000000000,10.0.0.0-10.255.255.255,10.0.0.0-10.255.255.255,9,7,tcp-rst-from-client,20,11,0,0,PA2.FL13,from-policy,0,0,N/A,0,0,0,0,1c563515-f2e7-227d-dc61-76f72a114777,0,0,2024-12-02T15:18:02.940-05:00,infrastructure,networking,network-protocol,2,“has-known-vulnerability,tunnel-other-application,pervasive-use”,msrpc,untunneled,no,no,0,NonProxyTraffic, ![Grok-Debugger](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/8/3/83500576dc8b793f4cfc9128e21adf15cfc2640d.png) The problem appears to be in how the community forum software formats the text being pasted into it: ![Graylog-Community](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/3/4/34cd279bf1aa4b53e9765a4fd88282e04575c6e8.png) We noticed that if we test the text in Grok Debugger by copying from the left side entry pane, it will work successfully - but if we test by copying from the right side preview pane, it fails at natsport. ---
**Author:** ![patrickmann](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/patrickmann/32/9091_2.png) [@patrickmann](https://community.graylog.org/u/patrickmann)\ **Post date:** [January 9, 2025, 9:15am UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/16 "2025-01-09T09:15:18Z")
Good catch - the regular double-quotes got converted to left and right double-quotes. However, I am still unable to get beyond `natsport`. The `NUMBER` pattern consumes the leading `0` of `0x401a` so following patterns fail to match. Drew alrady pointed this out earlier: [Use of ${QUOTEDSTRING} causes Graylog to seize - Part II - #5 by drewmiranda-gl](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/5) The data simply does not match the pattern. Your screenshot of grokdebugger cuts off the interesting fields. Do you actually see `natdport` and subsequent fields being assigned values on the right side? ---
**Author:** ![marziglt](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@marziglt](https://community.graylog.org/u/marziglt)\ **Post date:** [January 9, 2025, 4:07pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/17 "2025-01-09T16:07:46Z")
> Do you actually see `natdport` and subsequent fields being assigned values on the right side? Yes - if we use our original sources for the pattern and sample (normally kept in a text file via Notepad++), the entire pattern groks successfully, completely, and accurately in both Graylog _Test with Sample Data_ and in [Grok Debugger](https://grokdebugger.com/). If we copy the pattern/sample text from the community forum page, it fails on Grok Debugger after _natsport_. ![Grok-Debugger3](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/1/a/1a52225c9d5d4ccfe9ecf10d775d35104f8db717.png) Is there perhaps a way to send/upload a text file? ---
**Author:** ![patrickmann](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/patrickmann/32/9091_2.png) [@patrickmann](https://community.graylog.org/u/patrickmann)\ **Post date:** [January 9, 2025, 4:40pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/18 "2025-01-09T16:40:04Z")
You could file an issue in our public github repo [GitHub - Graylog2/graylog2-server: Free and open log management](https://github.com/Graylog2/graylog2-server) I notice that the results I see diverge somewhere before `outboundif`, because that is being assigned the timestamp that should be matched with `unknown_time`. > { > “logsrc”: “panorama”, > “num”: 1, > “receive\_time”: “2024/12/02 15:18:02”, > “serial”: 24680993766, > “type”: “TRAFFIC”, > “subtype”: “end”, > “unknown\_num”: 2897, > “time\_generated”: “2024/12/02 15:18:02”, > “src”: “10.88.151.8”, > “dst”: “10.99.101.17”, > “natsrc”: “0.0.0.0”, > “natdst”: “0.0.0.0”, > “rule”: “inernal-in\_37”, > “srcuser”: “msrpc-base”, > “dstuser”: “vdiv1”, > “app”: “PROD.Internal”, > “vsys”: “PROD.DIVNET”, > “from”: “ethernet1/2”, > “to”: “ethernet1/1”, > “inbound\_if”: “default”, > “outboundif”: “2024/12/02 15:18:02”, > “logset”: 655765, > “unknown\_time”: 1, > “sessionid”: 61636, > “repeatcnt”: 49681, > “sport”: 0, > “dport”: 0, > “natsport”: 0 > } > ] ---
**Author:** ![marziglt](https://avatars.discourse-cdn.com/v4/letter/m/4da419/32.png) [@marziglt](https://community.graylog.org/u/marziglt)\ **Post date:** [January 9, 2025, 8:31pm UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/19 "2025-01-09T20:31:45Z")
> You could file an issue in our public github repo [GitHub - Graylog2/graylog2-server: Free and open log management](https://github.com/Graylog2/graylog2-server) We will look into the feasibility of doing this. > I notice that the results I see diverge somewhere before `outboundif`, because that is being assigned the timestamp that should be matched with `unknown_time`. Don’t know what else to tell ya there. We only get those results when copying the pattern/sample text from the community message board posts here. When we use our own local copy of the pattern/sample text, it works fine - both in Graylog and on Grok Debugger. We’ve been successfully/accurately parsing over 10,000 logs per minute with the truncated version of it for weeks now. The trouble only starts when we try to grok a quoted string, and even that works on the SYSTEM logs which only come in once per minute. It seems like Drew’s idea about the quoted string grok causing massive CPU utilization (due to backtracking or some other issue) is the prevailing theory. When Graylog seizes, we notice via the _top_ command that CPU utilization jumps to over 400%. Then when we remove the grok and restart the service, it appears to empty out a process buffer that built up during the seizure, then returns to a normal level of CPU utilization (around 25-40%). We just don’t know how to grok the quoted strings from these logs at this volume without the seizures happening. ---
**Author:** ![patrickmann](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/patrickmann/32/9091_2.png) [@patrickmann](https://community.graylog.org/u/patrickmann)\ **Post date:** [January 10, 2025, 8:49am UTC](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564/20 "2025-01-10T08:49:13Z")
We use _io.krakens.grok.api_. QUOTED\_STRING is defined [here](https://github.com/thekrakken/java-grok/blob/901fda38ef6d5c902355eb25cff3f4b4fc3debde/src/main/resources/patterns/patterns#L19) as > (?\>(?\“(?\>\.|[^\”]+)+“|”"|(?\>‘(?\>\.|[^\’]+)+‘)|’'|(?\>`(?>\\.|[^\\`]+)+`)|``)) Maybe you can use a simpler pattern to match the specific strings you encounter in your TRAFFIC logs. [Next page](https://community.graylog.org/t/use-of-quotedstring-causes-graylog-to-seize-part-ii/34564.md?page=2)