Hi Guys,
i have a correct incoming JSON transfer to my Graylog instance.
my msg field has multiple words like “ET POLICY Vulnerable Java Version 1.7.x Detected”
i want to add quick values to my dashboard using this msg field.
But i get not the full text message as value, i get each word as value on my dasboard
If you need more informations let me know.
I hope somebody can help me.
How are you ingesting these messages?
Which extractors or pipeline rules are you using to process these messages?
Also, please provide some example messages.
I have an csv output of snort IDS and use a unix socket to transmit the infos to the tdagent/fluentd, which transmit the correctly JSON format to graylog (including the required field timestamp and source-host).
Because of the csv output i have no pipelining rules in graylog. The JSON goes dedicated from my tdagent to graylog.
msg examples:
-“ICMP test detected”
-“Consecutive TCP small segments exceeding threshold”
-“ICMP test detected”
There auch thousands of different messages. I want as example display the top 5 of messages. Maybe its the message “ICMP test detected”
Do you have an example?
2018-02-01 07:53:50.161515442 +0100 fluentd.var.log.snort.snort.csv: {"src":"192.168.xxx.xx","dst":"xxx.xxx.xxx.xxx","msg":"ET POLICY Vulnerable Java Version 1.7.x Detected","sport":null,"dport":null,"proto":"TCP","sig_id":"2014297","timestamp":"2018-02-01 07:53:50.161","source":"snrt001"}
And how are you processing that message in Graylog?
I have a stream input > msg field checked -> quick values -> add to dashboard
Or what do you exactly mean with processing?
If you want to use anything from that JSON payload, you either have to use a JSON extractor or a pipeline rule to parse the JSON and set the message fields accordingly.
Otherwise it’s just opaque text for Graylog.
Is this the necessary way, if i have the message field already “set” via transmission of the mentioned JSON-string?
Because the fields are correctly set in graylog…
You didn’t provide any details about the transmission, ingest, or the input configuration in Graylog…
Thats my input configuration of my Transmission client:
<source >
rotate_wait 5
@type tail
path /var/log/snort/snort.csv
pos_file /var/log/td-agent/tmp/access.log.1.pos
tag fluentd.*
<parse >
@type csv
keys src,dst,msg,sport,dport,proto,sig_id
time_key time
keep_time_key true
</parse>
enable_watch_timer True
refresh_interval 60
</source>
<filter fluentd.**>
enable_ruby true
@type record_transformer
<record>
timestamp ${time.strftime("%Y-%m-%d %H:%M:%S.%L")}
source "#{Socket.gethostname}"
</record>
</filter>
<match fluentd.**>
<store >
protocol tcp
add_msec_time true
@type gelf
host graylog-gelfXYZ.com
use_record_host true
port 12201
</store>
<store >
output_type json
@type stdout
@log_level warn
</store>
@type copy
</match>
The JSON, that is going to graylog is that here:
{"src":"192.168.xxx.xx","dst":"xxx.xxx.xxx.xxx","msg":"ET POLICY Vulnerable Java Version 1.7.x Detected","sport":null,"dport":null,"proto":"TCP","sig_id":"2014297","timestamp":"2018-02-01 07:53:50.161","source":"snrt001"}
in Graylog i defined a stream with this as input rule:
^.*snrt.*$
I hope this information helps.
here are the fields in a message in detail…
The “message” field is analyzed by default, which means it’s tokenized and each token is indexed. That’s why you get “single words” when running Quick Values on that field.
See https://www.elastic.co/guide/en/elasticsearch/guide/2.x/analysis-intro.html for details.
You could copy the “message” field into another field (either by using a Copy Input extractor or using the processing pipeline) which is not analyzed.
Also make sure to create a custom mapping for that field:
http://docs.graylog.org/en/2.4/pages/configuration/elasticsearch.html#custom-index-mappings
Thanks a lot. I will try this…
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.