Dead end in pipeline rules doing DNS TXT lookups

Graylog Central (peer support)
1

Hello,

Using Graylog 4.2.5.

I am writing what seemed to be a simple pipeline rule which must do a DNS TXT lookup.

I have defined a data adapter, cache and lookup table and they work.

However, the data it returns looks like this:

{
“single_value”: null,
“multi_value”: {
“raw_results”: [
{
“value”: “The string I want”,
“dns_ttl”: 14400
}
]
},
“string_list_value”: [
“The string I want”
],
“has_error”: false,
“ttl”: 14400000
}

And I can’t find a way to process this from a pipeline rule.

If I do this,

let ans = lookup(thetable, key)

But I can’t find a way at all to parse the result. raw_results is an array as far as I understand, and indeed if I do:

set_fields(ans)

This is the field created by Graylog:

[{“value”:“The string I want”,“dns_ttl”:14400}]

Is there any way to address this? Or have I hit a dead end or bug?

Thanks!

2

I managed an awful hack to make it work. At least I can get things done, but I shouldn’t need this.

Assuming that most of the time I am going to receive a single line as an answer to my TXT DNS queries, I have done this:

(please keep a barf bag handy!)

let awful = join(ans.raw_results)
// now awful is a string, so I can parse it.

let answer_fields = grok(“TxtDnsAnswer\{value=ANDNOWMYINTENDEDGROKPATTERN”…)

As I said, this kludge allows me to get stuff done but it’s ugly!

Should I report this as an issue, like “DNS TXT lookups are almost unusable”?

3

Hello @borjam

I’m not sure and it would help for better clarity what’s going on.
What I do know is:

  1. Your using a data adapter, cache and lookup table and they work.

  2. You receive a message in this format.

“single_value”: null,
“multi_value”: {
“raw_results”: [
{
“value”: “The string I want”,
“dns_ttl”: 14400
}
]
},
“string_list_value”: [
“The string I want”
],
“has_error”: false,
“ttl”: 14400000
}
  1. You created a pipeline.

Perhaps there might be a different way going about getting the end results then what you doing right now but I’m not 100% sure to offer any advice for different options. What does the message look like prior to using you Data adaptor? Is it in some type of JSON format being ingested.
DNS TXT lookup are you referring to DNS IP lookup or some type of DNS PTR?

4

Thank you. The data adapter, cache and look up table work without issues.

I am using Team Cymru’s ASN information service querying it via DNS.

https://team-cymru.com/community-services/ip-asn-mapping/#dns

For example, if I want to know about the 8.8.8.8 IP address:

;; QUESTION SECTION:

;8.8.8.8.origin.asn.cymru.com. IN TXT

;; ANSWER SECTION:

8.8.8.8.origin.asn.cymru.com. 14400 IN TXT "15169 | 8.8.8.0/24 | US | arin | 1992-12-01"

The response is a single line I can parse. Parsing it is trivial using, for instance, Grok. But turns out the object returned by the lookup() function looks like I showed above, a one element array. And I don´t see any way to access array elements from the pipeline rule language.

I found out that I can kinda “collapse” it into a single text line and at least I can parse it from there using Grok.

After collapsing it, this is the result which at least I can parse.

TxtDnsAnswer{value=15169 | 8.8.8.0/24 | US | arin | 1992-12-01., dnsTTL=14400}

For some addresses I can receive more than one line, like this example, but it is irrelevant for my purposes for now.

;1.38.239.216.origin.asn.cymru.com. IN TXT

;; ANSWER SECTION:

1.38.239.216.origin.asn.cymru.com. 14400 IN TXT "15169 | 216.239.38.0/24 | US | arin | 2000-11-22"

1.38.239.216.origin.asn.cymru.com. 14400 IN TXT "15169 | 216.239.32.0/19 | US | arin | 2000-11-22"

The first line is enough for me, as it shows the most specific route.

And the only way I found to parse this is to do this ugly hack:

    let query = concat(query, ".origin.asn.cymru.com");
    let ans = lookup("cymruasn", query);
    let cosa = join(ans.raw_results);
    let ansfields = grok("TxtDnsAnswer\\{value=%{INT:asn} \\| %{NOTSPACE:cidraddr} \\| %{NOTSPACE:country} \\| %{NOTSPACE:ncc} .*", cosa, true);

It works. For now it’s enough for my nefarious purposes, but I wonder whether I found a bug or oversight in the implementation of the TXT DNS lookup data adapter.

Thanks!

5

Hello
Thank you for the clarity.

To be honest I’m not sure. From what I read this should work. I personal haven’t used Data adaptor for that purpose. My main Go-To is regex with an Extractor but I did find this post, perhaps it might help.

If you believe its a bug you could ask here.

6

Thank you. Seems the guy from that post hit the same problem, no way to access elements in a multi value. And the problem for the TXT DNS queries is, even if there is a single answer it will be an array/mutli value.

I will file a bug report about an incomplete feature. The rule language should have some way to access those multi values.

1 Like
7

OK found a clean solution!!

Turns out there is a function called lookup_string_list() which will take a look up table and a search key as an argument and it will return a list. It is intended to be used for queries that will return multiple results. Not to confuse with lookup(), which will expect a single result with multiple fields (a map).

Once you have the list, it is possible to use the first_non_null() function which will effectively return
the first element.

So, my ugly code would get better like this:

// This look up function can return multiple results in a list, _or_ a one element list
answer_list = lookup_string_list("table", key);
// Take the first element from the list
answer = first_non_null(answer_list);
// Convert it to a string (in case it was a map you shouldn't do this)
line = to_string(answer);
// And now we can parse, grok, whatever.

So if I lookup, say, 8.8.8.8.origin.asn.cymru.com, answer would be:
“15169 | 8.8.8.0/24 | US | arin | 1992-12-01.”

Which was exactly the intended result.

Now, still wondering how to access multiple elements in a list. Not a pressing need but it might be.

And it seems that first_non_null consumes them. Now it would be great to have some kind of iterator.

2 Likes
8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.