Key Value Pairs

[Cyver (Oliver)]

Hi,

is there any default functionality in terms of key value pairs extraction?

Why I am asking - I found some fields in my graylog instance which were NOT defined in any extractor. So i deleted the whole INPUT, created a new (empty) one without any extractors and … suprise … all the fields are extracted again (looks like KVP)? Its a little bit spooky for me

I am running latest Graylog 2.4.0 Version cluster with 3 nodes and 5.4.x ES.

Many Thanks & BR
Oliver

[jochen]

What type of input have you been using?
What is the full configuration of the input?
What kind of log messages has Graylog been receiving?

I found some fields in my graylog instance which were NOT defined in any extractor.

Please provide some examples.

[Cyver (Oliver)]

Input typ: Syslog UDP

Full configuration:

allow_override_date: true
bind_address: 0.0.0.0
expand_structured_data: false
force_rdns: false
override_source:
port: 5001
recv_buffer_size: 262144
store_full_message: false

• static fields:

transportflag: direct
vendorflag: fortigate

log message example:

date=2018-01-15 time=16:14:05 devname=xxxxx devid=FGT1111111 logid=0000000013 type=traffic subtype=forward level=notice vd=root srcip=10.xx.xx.xx srcport=68 srcintf=“xxxxx-xxxxx” dstip=xx.xx.xx.xx dstport=67 dstintf=“xxx-xxxxx” sessionid=0000000 proto=17 action=deny policyid=0 policytype=policy dstcountry=“Reserved” srccountry=“Reserved” trandisp=noop service=“DHCP” duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat=“unscanned” crscore=30 craction=131072 crlevel=high

Its default Fortigat Syslog format.

which is

Received by
GLOB Fortigate UDP on 12ef998f / nw-glog3
Stored in index
graylog_5

Input Extractors:

Extractors of GLOB Fortigate UDP
Extractors are applied on every message that is received by this input. Use them to extract and transform any text data into fields that allow you easy filtering and analysis later on. Example: Extract the HTTP response code from a log message, transform it to a numeric field and attach it as http_response_code to the message.

Find more information about extractors in the documentation.
Add extractor
Start by loading a message to have an example to work on. You can decide whether to load a recent message received by this input, or manually select a message giving its ID.

Get started

Configured extractors
This input has no configured extractors.

br
oliver

[jochen]

Its default Fortigat Syslog format.

Yes, these are parsed by default:

Add support for Cisco and FortiGate syslog messages

[Cyver (Oliver)]

Many THX Jochen!

And it seems I am hitting the same bug as described in bottom - url field in syslog will create a lot of unusable fields (until maximum fields limit in ES reached and ES stops … yihaa). Any chance to disable this functionality? further … we are using different field names … so no need for using default kvp extraction…

Thx & BR
Oliver

[jochen]

Any chance to disable this functionality?

No, but you could use a Raw/Plaintext UDP input and extract the information you want via some extractors or pipeline rules.

[Cyver (Oliver)]

Plaintext UDP works as exprected - many thanks

This default behavior also effects GELF Inputs which receives Fortinet syslogs from collectors?

BR

[jochen]

This default behavior also effects GELF Inputs which receives Fortinet syslogs from collectors?

No, only Syslog TCP and Syslog UDP inputs.