# Input Extractor Not Capture All Logs **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [July 29, 2024, 5:44pm UTC](https://community.graylog.org/t/input-extractor-not-capture-all-logs/33136 "2024-07-29T17:44:04Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![secdoc](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/secdoc/32/14813_2.png) [@secdoc](https://community.graylog.org/u/secdoc)\ **Post date:** [July 29, 2024, 5:44pm UTC](https://community.graylog.org/t/input-extractor-not-capture-all-logs/33136/1 "2024-07-29T17:44:04Z")
Before you post: Your responses to these questions will help the community help you. Please complete this template if you’re asking a support question. **Don’t forget to select tags to help index your topic!** **1. Describe your incident:** Currently have an input from OPNsense FW. The filterlog is parsing with the exception of the the following example log: ```auto <134>Jul 29 12:26:15 gambit. *****.***** filterlog[40156]: 84,,,5ca69255279ee5173e47af3fb24bacef,bridge0,ip-option,pass,in,4,0xc0,,1,59085,0,DF,2,igmp,36,192.168.200.254,224.0.0.1,datalength=12 ``` This is what it looks like not parsed: ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/d/a/daa2a2f99722c00202f2f8907c24ac823c7a55dc.png) Other filterlogs such as the following parse without issue: ```auto <134>Jul 29 12:26:15 gambit. *****.***** filterlog[40156]: 85,,,9f96d956119c25145fc2ce221237f3a5,bridge0,match,pass,out,4,0x0,,62,50557,0,DF,6,tcp,60,99.137.185.156,172.67.216.12,60624,443,0,S,2282301637,,64240,,mss;sackOK;TS;nop;wscale ``` Here is an example of what it looks like properly parsed: ![image](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/4/1/413da003bc1c45660fce1c2a15cbe01126bd4afb.png) Here is the input extractor that I have currently: ```auto { "extractors": [ { "title": "OPNsense filterlog: IPv4 TCP", "extractor_type": "grok", "converters": [], "order": 0, "cursor_strategy": "copy", "source_field": "message", "target_field": "", "extractor_config": { "grok_pattern": "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} filterlog\\[%{NUMBER:pid}\\]: %{NUMBER:rule_number},%{DATA:sub_rule},%{DATA:anchor},%{DATA:tracker},%{DATA:interface},%{DATA:reason},%{DATA:action},%{DATA:direction},4,%{DATA:tos},%{DATA:ecn},%{NUMBER:ttl},%{NUMBER:id},%{NUMBER:offset},%{DATA:flags},%{NUMBER:protocol_id},tcp,%{NUMBER:length},%{IP:src_ip},%{IP:dst_ip},%{NUMBER:src_port},%{NUMBER:dst_port},%{NUMBER:data_length},%{DATA:tcp_flags},%{NUMBER:sequence_number},%{DATA:ack_number},%{NUMBER:window},%{DATA:urg},%{GREEDYDATA:tcp_options}", "named_captures_only": true }, "condition_type": "string", "condition_value": "filterlog" }, { "title": "OPNsense filterlog: IPv4 UDP", "extractor_type": "grok", "converters": [], "order": 1, "cursor_strategy": "copy", "source_field": "message", "target_field": "", "extractor_config": { "grok_pattern": "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} filterlog\\[%{NUMBER:pid}\\]: %{NUMBER:rule_number},%{DATA:sub_rule},%{DATA:anchor},%{DATA:tracker},%{DATA:interface},%{DATA:reason},%{DATA:action},%{DATA:direction},4,%{DATA:tos},%{DATA:ecn},%{NUMBER:ttl},%{NUMBER:id},%{NUMBER:offset},%{DATA:flags},%{NUMBER:protocol_id},udp,%{NUMBER:length},%{IP:src_ip},%{IP:dst_ip},%{NUMBER:src_port},%{NUMBER:dst_port},%{NUMBER:data_length}", "named_captures_only": true }, "condition_type": "string", "condition_value": "filterlog" }, { "title": "OPNsense filterlog: IPv4 ICMP", "extractor_type": "grok", "converters": [], "order": 2, "cursor_strategy": "copy", "source_field": "message", "target_field": "", "extractor_config": { "grok_pattern": "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} filterlog\\[%{NUMBER:pid}\\]: %{NUMBER:rule_number},%{DATA:sub_rule},%{DATA:anchor},%{DATA:tracker},%{DATA:interface},%{DATA:reason},%{DATA:action},%{DATA:direction},4,%{DATA:tos},%{DATA:ecn},%{NUMBER:ttl},%{NUMBER:id},%{NUMBER:offset},%{DATA:flags},%{NUMBER:protocol_id},icmp,%{NUMBER:length},%{IP:src_ip},%{IP:dst_ip},%{GREEDYDATA:icmp_type}", "named_captures_only": true }, "condition_type": "string", "condition_value": "filterlog" }, { "title": "OPNsense filterlog: IPv4 Other Protocols", "extractor_type": "grok", "converters": [], "order": 3, "cursor_strategy": "copy", "source_field": "message", "target_field": "", "extractor_config": { "grok_pattern": "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} filterlog\\[%{NUMBER:pid}\\]: %{NUMBER:rule_number},%{DATA:sub_rule},%{DATA:anchor},%{DATA:tracker},%{DATA:interface},%{DATA:reason},%{DATA:action},%{DATA:direction},4,%{DATA:tos},%{DATA:ecn},%{NUMBER:ttl},%{NUMBER:id},%{NUMBER:offset},%{DATA:flags},%{NUMBER:protocol_id},%{WORD:protocol},%{NUMBER:length},%{IP:src_ip},%{IP:dst_ip},%{GREEDYDATA:additional_info}", "named_captures_only": true }, "condition_type": "string", "condition_value": "filterlog" }, { "title": "OPNsense Suricata", "extractor_type": "grok", "converters": [], "order": 4, "cursor_strategy": "copy", "source_field": "message", "target_field": "", "extractor_config": { "grok_pattern": "%{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} suricata\\[%{NUMBER:pid}\\]: %{DATA:info} \\[Priority: %{NUMBER:priority}\\] \\{%{DATA:protocol}\\} %{IP:src_ip}:%{NUMBER:src_port} -> %{IP:dst_ip}:%{NUMBER:dst_port}", "named_captures_only": true }, "condition_type": "string", "condition_value": "suricata" }, { "title": "SourceIP geo-lookup-country", "extractor_type": "lookup_table", "converters": [], "order": 5, "cursor_strategy": "copy", "source_field": "src_ip", "target_field": "src_country", "extractor_config": { "lookup_table_name": "geo-lookup-country" }, "condition_type": "none", "condition_value": "" }, { "title": "DestIP geo-lookup-country", "extractor_type": "lookup_table", "converters": [], "order": 6, "cursor_strategy": "copy", "source_field": "dst_ip", "target_field": "dst_country", "extractor_config": { "lookup_table_name": "geo-lookup-country" }, "condition_type": "none", "condition_value": "" } ], "version": "5.0.0" } ``` I am looking for some Ideas on what either I am doing incorrectly or a way to improve the extractor. **2. Describe your environment:** - OS Information: CPU: AMD Ryzen 9 6900HX (16) @ 3.30 GHz GPU: AMD Rembrandt @ 2.40 GHz [Integrated] Memory: 12.24 GiB / 28.15 GiB (43%) SHELL: bash 5.1.16 OS: Ubuntu jammy 22.04 x86\_64 KERNEL: Linux 5.15.0-112-generic - Package Version: ```auto graylog-6.0-repository/stable,now 1-1 all [installed] graylog-server/stable,now 6.0.4-1 amd64 [installed] libmongoc-1.0-0/jammy,now 1.21.0-1build1 amd64 [installed,automatic] libmongocrypt0/jammy,now 1.3.0-1ubuntu1 amd64 [installed,automatic] mongodb-database-tools/jammy,now 100.9.5 amd64 [installed,upgradable to: 100.10.0] mongodb-mongosh/jammy,now 1.9.0 amd64 [installed,upgradable to: 2.2.14] mongodb-org-database-tools-extra/jammy,now 6.0.16 amd64 [installed,automatic] mongodb-org-database/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.16] mongodb-org-mongos/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.16] mongodb-org-server/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.16] mongodb-org-shell/jammy,now 6.0.16 amd64 [installed,automatic] mongodb-org-tools/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.16] mongodb-org/jammy,now 6.0.6 amd64 [installed,upgradable to: 6.0.16] opensearch/stable,now 2.15.0 amd64 [installed] ``` - Service logs, configurations, and environment variables: n/a **3. What steps have you already taken to try and solve the problem?** I have tried adjusting the extractor with no success… **4. How can the community help?** I am looking for some Ideas on what either I am doing incorrectly or a way to improve the extractor. **Helpful Posting Tips:** [Tips for Posting Questions that Get Answers](https://community.graylog.org/t/tips-for-posting-questions-that-get-answers/21828) [Hold down CTRL and link on link to open tips documents in a separate tab] ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [August 12, 2024, 5:44pm UTC](https://community.graylog.org/t/input-extractor-not-capture-all-logs/33136/2 "2024-08-12T17:44:43Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.