Correct parsing of list/array values, e.g. from python

Hello there,
I made a neat little discovery working with arrays (or lists, to stay in python wording). Hoping it’s not cold coffee for most of you, I decided to share it.

More than once, I had problems correctly searching lists of elements in graylog, especially when trying to use list values in dashboards.
Finally, I created three scenarios of how to process list values, all based on a simple python script[1].

First, I sent in the list from python unchanged and did not change it. The result is as follows:

cve_list count()
[CVE-2022-1234,CVE-2022-2345,CVE-2022-3456,CVE-2022-4567] 1

As you can see, this is quite inconvenient, no statistics will work, nor can you search for a single CVE.

What you can do is splitting the incoming list at “,” with a pipeline rule[2] (that will also trim the leading and trailing brackets). This will generate single values, allowing graylog to show the following:
grafik
As you can see, the values are split now. You can search for any single of them, while also having statistics work (e.g., sending in a log containing only CVE-2022-1234 and CVE-2022-3456 will add 1 to their respective counters, leaving the others unchanged).

Although it looks a bit fiddly, it will work for any list (not containing “,” in any of its values, since this is used by graylog to list them).
What you could also do is joining the list’s elements before sending them in (with a delimiter not used in its values, obviously) to utilize the split() function in a pipeline rule. But that is up to you to find out :wink:

I hope this is helpful, feel free to share your thoughts.

Best regards
Tim

[1]

python code for sending logs
import logging
import graypy

logger = logging.getLogger('gelf_tester')
logger.addHandler(graypy.GELFTCPHandler(host='<your graylog's IP address goes here', port=<its input port goes here>, debugging_fields:False))

cve_list = ['CVE-2022-1234','CVE-2022-2345','CVE-2022-3456', 'CVE-2022-4567']
logger.critical(msg='Hello, this is a test message', extra={"cve_list":cve_list, "program":"gelf_tester"})"program":"gelf_tester"})

[2]

rule "cve_list_splitted"
when
    has_field("cve_list")
then
    set_field(field:"cve_list_splitted", value:split(pattern:"\",\"", value:substring(to_string($message.cve_list), 2, -2)));
end
1 Like

@tbe

Thanks for sharing :smiley:

1 Like