Migrate Dashboards from Spl__k to Graylog

[infosecian (Dennis)]

1. Describe your incident:
We try to switch over from Spl__k to Graylog. We onboarded the logsources. We try to adopt the dashboards and queries, but failed at certain points:

• World map: derive from an IP address the country and display it on a world map.
• Show number of logs (easy), group by Day of Week (sun, mon, tue, wed, …) - field not found
• Show number of logs (still easy), group by Hour (0,1,2,3,…23) - all values are empty
• Show data table (done) with trendline/sparkline - have not found a Graylog equivalent for that
• Show data table grouped by concatenation of two fields - have not found that or can’t this be done during search time? Only found somethin’ with Pipelines.
• Show data table of top IPs and put the DNS name next to them (lookup on the fly)
• Show a statistic table with sum of bytes (in and out has to sum up as well) and number of entries (=connections) per IP

This shows one (incomplete) Dashboard:

2. Describe your environment:

• OS Information: Ubuntu 22.04.2 LTS
• Package Version: 5.0.8-1

3. What steps have you already taken to try and solve the problem?
Read the documentation, explanation of Query Language as well as the creation of widgets

4. How can the community help?
In general, which of the above points are applicable with Graylog? Hopefully some hints (keywords). Would be splendid if it can be shown that most of the created insights can also created by or with Graylog.

[gsmith]

Hey @infosecian

Some Widgets as you know can be done, if it cant a pipeline would be needed to create that field so the widget can be built. There are extractors if you wish but a pipeline would be you best bet for modifying the logs to get what you want.

• Pipelines

[infosecian (Dennis)]

Hello @gsmith ,

I try to understand the pipeline feature. As far as I understand the pipelines are in place like this:
logs > extractors > pipelines > OpenSearch < search < dashboard

Does this mean that only kv pairs can be queried that were already stored in OpenSearch? No way to generate fields during the runtime? This increases the data size and creates redundant information.

example:
src-ip: 1.2.3.4
src-port: 43123
dest-ip: 10.83.13.81
dest-port: 443

additional information:
socket: 10.83.13.81:443 ← redundant
dest-dns: webserver01.int.lan ← may change over time, pro and cons to persist this
src-dns: tso-119-146.customers.provider.com ← may change over time, pro and cons to persist this
src-geoloc: 53.69054950,8.20420349302 ← pipelining
src-city: ABC ← pipelining
dest-geoloc: 52.92349023,7.9301293921 ← pipelining
dest-city: XYZ ← pipelining

I guess I start with the geo location part. The missing “create new columns on the fly” irritates me, since this is also very common with databases.

[ihe]

You can use Decorators to add stuff while viewing. You can not search by those values though.

For the Geo-IP widgets the steps are those:

1. put a geodb-database on your Graylog-Host with sufficient permissions for Graylog to read it
2. create a dataadapter to read that file. Test it here!
3. create a cache for the queries
4. connect dataadapter and cache in a lookuptable
5. build a pipeline doing queries on the lookuptable for some of your field
6. save those results into an extra field or add them via decorator