Description of your problem
Graylog can be hard to use. I have had much success with coupling ELK. Part of the success is that there were tonnes of tutorials online on how to get started. DigitalOcean published one just late 2020. Before then, I had used nsm-script (on github e.g. logstash-dfir/bro-ids_logstash.conf at master · timmolter/logstash-dfir · GitHub) containing tonnes of grok patterns to drop into logstash.
Part of the problem with graylog is that logstash is now useless as rsyslog takes over it. And for the logs and indices, I now need write regex and grok to create graphical output followed be a dashboard (a collection of several outputs).
Of course, graylog has its merit. It’s tightly coupled; hence ease of maintenance is assured.
I cannot create graphs and dashboards from my logs; see sample log messages below
Description of steps you’ve taken to attempt to solve the issue
Several of the limited information on this forum, marketplace, Internet search and so on have been processed to make a graph and dashboard but no luck.
Environmental information
Graylog 4.0.7+c3e766c
Operating system information
on (OpenJDK BSD Porting Team 1.8.0_292 on FreeBSD 13.0-RELEASE-p1)
Package versions
Graylog 4.0.7+c3e766c
###Code/log
As a practical example, I have the few messages selected after runing a search with a few keywords.
filterlog[75974]: 68,,,12004,igb0,match,block,in,4,0x0,,64,0,0,DF,17,udp,201,10.1.1.1,255.255.255.255,42486,7437,181
filterlog[75974]: 173,,,1612113360,igb0,match,pass,in,4,0x0,,43,45698,0,DF,6,tcp,60,117.146.54.17,13.97.34.34,38076,4002,0,S,2079802503,,42340,,mss;sackOK;TS;nop;wscale
suricata[46342]: [1:2210029:2] SURICATA STREAM ESTABLISHED invalid ack [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} 188.46.34.90:4002 -> 36.137.218.2:4001
filterlog[75974]: 216,,,1614348250,igb2,match,pass,in,4,0x0,,64,41762,0,none,17,udp,64,13.97.34.34,91.217.137.37,62691,53,44
suricata[46342]: [1:2210045:2] SURICATA STREAM Packet with invalid ack [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} 85.152.174.181:36907 -> 188.46.34.90:4002
suricata[89495]: [1:2221034:1] SURICATA HTTP Request unrecognized authorization method [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} 188.46.34.90:13933 -> 52.21.162.196:80
1 2020-03-20T15:41:12.371725+00:00 check.litewave.nl rsyslogd 7252 file '/var/log/maillog'[6] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2001.0 try https://www.rsyslog.com/e/2027 ]
suricata[25365]: {"timestamp": "2020-12-21T09:45:45.410870+0800", "flow_id": 1334437720184452, "in_iface": "pppoe0", "event_type": "dns", "src_ip": "183.67.34.60", "src_port": 52134, "dest_ip": "216.239.36.10", "dest_port": 53, "proto": "UDP", "dns": {"version": 2, "type": "answer", "id": 52445, "flags": "8400", "qr": true, "aa": true, "rrname": "_domainkey.gmail.com", "rrtype": "A", "rcode": "NOERROR", "authorities": [{"rrname": "gmail.com", "rrtype": "
1 2020-10-01T17:09:17.497909+00:00 check.litewave.nl sm-mta 57473 091H9GsA057473: ruleset=check_rcpt, arg1=<r19772744@gmail.com>, relay=[51.132.251.70], reject=550 5.7.1 <r19772744@gmail.com>... Relaying denied. IP name lookup failed [51.132.251.70]
suricata[46342]: [1:2260002:1] SURICATA Applayer Detect protocol only one direction [Classification: Generic Protocol Command Decode] [Priority: 3] {TCP} 13.97.34.36:25 -> 203.159.80.219:55700
1 2021-08-01T12:25:01.453462+00:00 sv.hdns.lo dnsdist 12042 Marking downstream 91.217.137.37:53 as 'up'
1 2020-03-05T05:02:18.413455+00:00 mail.milzone.org dovecot 75430 imap-login: Login: user=<bjames@msn.com>, method=PLAIN, rip=202.52.36.51, lip=192.168.3.2, mpid=89538, TLS, session=<ZUv0cBSgHTXKNCQz>Are there goto examples for making graphs and dashboard from each? The templates on this forum are not detailed for a beginner. There are hardly free and effective plugins available in grayloy marketplace. There are scanty blogposts online. And so on. I am almost going back to ELK (here is one good documenttation - How To Install Elasticsearch, Logstash, and Kibana (Elastic Stack) on Ubuntu 20.04 | DigitalOcean). One problem with ELK is installing matching versions of Elasticsearch, Kibana and Logstash can sometimes be a herculean task on *BSD. If one gets them right, there are tonnes of logstash templates - *beat(filebeat/etc) - to use.
I had several dashboards providing great insights on our business network in the past years. Today, our business intelligence is practically zero. We have only got Graylog crunching logs from our firewall and /var/logs dir with not meaningful information. Please assist.