Extractor works in test but not on message

Graylog Central (peer support)
1

All my extractors stopped working, and I don’t get it. Please help!

  "extractors": [
    {
      "title": "pfSense filterlog: IPv6 TCP/UDP/ICMPv6",
      "extractor_type": "regex",
      "converters": [
        {
          "type": "csv",
          "config": {
            "column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,Protocol,ProtocolID,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength,TCP Flags,SeqNum,ACK,Window,URG,Options"
          }
        }
      ],
      "order": 1,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "FilterData",
      "extractor_config": {
        "regex_value": "^filterlog:\\s+(.*)$"
      },
      "condition_type": "regex",
      "condition_value": "^filterlog:\\s+.*,(in|out),6,.*,([tTuU][cCdD][pP]|ICMPv6),.*$"
    },
    {
      "title": "pfSense filterlog: IPv4 TCP/UDP",
      "extractor_type": "regex",
      "converters": [
        {
          "type": "csv",
          "config": {
            "column_header": "RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength,TCP Flags,SeqNum,ACK,Window,URG,Options"
          }
        }
      ],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "FilterData",
      "extractor_config": {
        "regex_value": "^filterlog:\\s+(.*)$"
      },
      "condition_type": "regex",
      "condition_value": "^filterlog:\\s+.*,(in|out),4,.*[tTuU][cCdD][pP],.*$"
    }
  ],
  "version": "3.0.0"
}

Example data:

RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,Protocol,ProtocolID,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength,TCP Flags,Sequence Number,ACK,Window,URG,Options,,,,
7,,,1000000105,re1,match,block,in,6,0x00,0x456fe,1,UDP,17,32,fe80::b58d:86be:236f:4200,ff02::1:3,64183,5355,32,,,,,,,,,,
493,,,1527184618,re1,match,block,in,6,0x00,0xb39dc,64,UDP,17,51,fd12:19f1:239f:3a6f:643d:9d33:88f0:72ff,fd12:19f1:239f:3a6f::1,54500,53,51,,,,,,,,,,

RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,14,15,16,17,Protocol,ProtocolID,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength,TCP Flags,Sequence Number,ACK,Window,URG,Options
11,,,1000000107,re1,match,pass,in,6,0x00,0x00000,255,ICMPv6,58,32,fe80::f227:65ff:fedb:1d03,fd12:19f1:239f:3a6f::1,,,,,,,,,,,,,

RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength,TCP Flags,Sequence,umber,ACK,Window,URG,Options
113,,,1000011061,lo0,match,pass,in,4,0x0,,64,65150,0,none,17,udp,70,127.0.0.1,127.0.0.1,29598,53,50,,,,,,,

RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,Protocol,ProtocolID,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength,TCP Flags,Sequence Number,ACK,Window,URG,Options,,,,
7,,,1000000105,re1,match,block,in,6,0x00,0x456fe,1,UDP,17,32,fe80::bXXd:XXe:2XXf:4XX0,ff02::1:3,64183,5355,32,,,,,,,,,,
493,,,1527184618,re1,match,block,in,6,0x00,0xb39dc,64,UDP,17,51,fdXX::7XXf,fdXX::7XXf,54500,53,51,,,,,,,,,,

RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,14,15,16,17,Protocol,ProtocolID,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength,TCP Flags,Sequence Number,ACK,Window,URG,Options
11,,,1000000107,re1,match,pass,in,6,0x00,0x00000,255,ICMPv6,58,32,fe80::bXXd:XXe:2XXf:4XX0,fdXX::7XXf,,,,,,,,,,,,,

RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength,TCP Flags,Sequence,umber,ACK,Window,URG,Options
113,,,1000011061,lo0,match,pass,in,4,0x0,,64,65150,0,none,17,udp,70,127.0.0.1,127.0.0.1,29598,53,50,,,,,,,

RuleNumber,SubRuleNumber,Anchor,Tracker,Interface,Reason,Action,Direction,IPVersion,TOS,ECN,TTL,ID,Offset,Flags,ProtocolID,Protocol,Length,SourceIP,DestIP,SourcePort,DestPort,DataLength,TCP Flags,Sequence,umber,ACK,Window,URG,Options
117,,,1000011065,re1,match,pass,out,4,0x10,,54,0,0,DF,6,tcp,60,185.93.2.228,10.0.0.11,17238,80,0,S,1603370064,,32120,,mss;sackOK;TS;nop;wscale,
2

So do the csv fields have to match exactly in number?

If the message has less more or less fields, woudn’t it fill all fields from rule, and leave some empty or add remaining fields in the last one?

Edit:
I’ve split them up again by protocol version and type to match the csv fields exactly against the message.

Only ipv4 tcp extractor works, the rest doesn’t. I see no reason why they shouldn’t.

Is Graylog having quirks like this?

3

So, the extractors work on some messages but not on all, been at this for hours :weary:

4

Or maybe it just takes hours for changes to the extractors to start showing in the search?

Most of them just started working. I made a a new extractor for ICMPv6 at 17:53, let’s see how it takes.

5

Pffff…

pfSense filterlog: IPv6 ICMP

Metrics
6,196 total invocations since boot, averages: 1.07, 3.75, 3.53.
267 hits, 5929 misses

None of those 267 hits can be found in the logs. What gives?

6

Started working at around 22:00.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.