Issues with attempting to pull Fields from Message


#1

Maybe I am misunderstanding the functionality of Graylog (I hope not). I have set up GROK extractors for incoming logs, specifically this extractor:

%{DATA:source} %{DATA:dvc} Protocol: %{WORD:proto}, SrcIP: %{IP:src_addr}(, OriginalClientIP: %{DATA:client_ip})?, DstIP: %{IP:dst_addr}, SrcPort: %{INT:src_port}, dport: %{INT:DestPort}, TCPFlags: %{BASE16NUM:tcp_flags}, IngressInterface: %{DATA:in_iface}, EgressInterface: %{DATA:out_iface}, IngressZone: %{DATA:in_zone}, EgressZone: %{DATA:out_zone}, DE: %{DATA:engine}, Policy: %{DATA:policy}, ConnectType: %{WORD:conn_type}, AccessControlRuleName: %{DATA:ac_rule_name}, AccessControlRuleAction: %{WORD:ac_action},( Prefilter Policy: %{DATA:prefilter_policy},)?( Client: %{WORD:client},)? UserName: %{DATA:user}, InitiatorPackets: %{INT:init_packets}, ResponderPackets: %{INT:resp_packets}, InitiatorBytes: %{INT:init_bytes}, ResponderBytes: %{INT:resp_bytes},( NAPPolicy: %{DATA:nap_policy},)?( DNSQuery: %{DATA:dns_query},)?( DNSRecordType: %{DATA:dns_rec_type},)? DNSResponseType: %{DATA:dns_resp_type}, Sinkhole: %{DATA:sinkhole}, URLCategory: %{DATA:url_cat}, URLReputation: %{GREEDYDATA:url_rep}

for the log (Anonymized)

DATASOURCE IPSAlerts: Protocol: TCP, SrcIP: 10.255.255.255, OriginalClientIP: ::, DstIP: 23.255.255.255, SrcPort: 36803, DstPort: 443, TCPFlags: 0x0, IngressInterface: inside, EgressInterface: outside, IngressZone: E_Inside, EgressZone: E_Outside, DE: Primary Detection Engine (), Policy: ASA, ConnectType: End, AccessControlRuleName: INSIDE-->OUTSIDE, AccessControlRuleAction: Allow, Prefilter Policy: Unknown, UserName: No Authentication Required, Client: SSL client, ApplicationProtocol: HTTPS, InitiatorPackets: 21, ResponderPackets: 44, InitiatorBytes: 5838, ResponderBytes: 51769, NAPPolicy: Balanced Security and Connectivity, DNSResponseType: No Error, Sinkhole: Unknown, URLCategory: Unknown, URLReputation: Risk unknown, URL: 

What I want to do is have the field DestPort searchable so I can run it into a stream that pulls messages that I am attempting to filter as High Risk Ports.

The extractor outputs this when I set it up:

**DestPort**
**443**
ac_action
Allow
ac_rule_name
INSIDE-->OUTSIDE
client
SSL client, ApplicationProtocol: HTTPS, WebApplication: 
client_ip
::
conn_type
End
dns_resp_type
No Error
dst_addr
23.255.255.255
dvc
IPSAlerts:
engine
Primary Detection Engine ()
in_iface
inside
in_zone
E_Inside
init_bytes
902
init_packets
10
nap_policy
Balanced Security and Connectivity
out_iface
outside
out_zone
Outside
policy
ASA
prefilter_policy
Unknown
proto
TCP
resp_bytes
3778
resp_packets
8
sinkhole
Unknown
source
DEVICE
src_addr
10.255.255.255
src_port
50779
tcp_flags
0x0
url_cat
Unknown
url_rep
Risk unknown, URL: 
user
No Authentication Required

however, after all of this when i go to search Graylog does not identify any ports with DestPort (and variations.) I would appreciate your thoughts or help, I can provide more information for clarity.


(Jochen) #2

What’s in the logs of your Graylog and Elasticsearch nodes?
:arrow_right: http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

Also, has your Graylog node been receiving messages matching the extractor condition?
Extractors are only applied to new messages, so existing messages won’t be re-processed after they’ve been received and indexed.


#3

Thank Jochen, the issue is in fact that I hadn’t received enough new logs and was only querying the old stock. Thank you for your help.


(system) #4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.