Hi,
I’m very new to Graylog, ElasticSearch, ELK and so on - Until a week ago I had never used Logstash, Filebeats and so on, so please forgive me if this one is obvious.
I have set up a test VM with CentOS7 in order to evaluate Graylog.
I have been importing some log files from our Exchange environment, and it’s all going great - I have a filebeat/sidecar picking up logs just from a local folder (for testing purposes) and I have several streams, rules, lookup tables, adaptors, extractors etc and i’m almost happy with the data, except I can’t find how to do anything with the user_agent field.
My goal is to be able to report/filter on various properties of the user_agent, such as Browser version, OS type, etc.
I have tried using the API at http://www.useragentstring.com/pages/api.php using a data extractor and that works, but the results you get back are quite limited. I’ve been scouring the net and there’s a lot of references to ua-parser which uses regexes.yaml and this seems to be the “common” way of doing it - But I have very little coding skills and would prefer to keep this Graylog server as “out of the box” as possible.
ElasticSearch apparently has this built in (https://www.elastic.co/guide/en/logstash/current/plugins-filters-useragent.html) but for the life of me I can’t figure out how to use it.
I have tried adding to the filebeat config, but I get the error
level=error msg="[filebeat] Validation command output: Exiting: error initializing publisher: error initializing processors: the processor user_agent doesn't exist\n"
Filebeat config is as follows (obviously I got the error when it was uncommented)
# Needed for Graylog
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
filebeat.inputs:
- input_type: log
paths:
- /tmp/loginput/ex2016iis/*.log
type: log
output.logstash:
hosts: ["192.168.93.228:5000"]
path:
data: /var/lib/graylog-sidecar/collectors/filebeat/data
logs: /var/lib/graylog-sidecar/collectors/filebeat/log
processors:
- drop_event:
when:
regexp:
message: "^#"
# This doesn't work
# - user_agent:
# field: "user_agent_string"
If the ability is built in, i’d much rather do that, but i’ve really only got a vague understanding of how the whole stack fits together so far.
Any help or comments greatly appreciated