# Search through CSV Data adapter via Regex **URL:** \ **Category:** Graylog Central (peer support)\ **Created:** [October 8, 2022, 7:44pm UTC](https://community.graylog.org/t/search-through-csv-data-adapter-via-regex/26088 "2022-10-08T19:44:24Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![dmuensterer](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/dmuensterer/32/13602_2.png) [@dmuensterer](https://community.graylog.org/u/dmuensterer)\ **Post date:** [October 8, 2022, 7:44pm UTC](https://community.graylog.org/t/search-through-csv-data-adapter-via-regex/26088/1 "2022-10-08T19:44:24Z")
**1. Describe your incident:** I am trying to create a data adapter whose values I can lookup by using a regex pattern. Let’s say I have values like this in my CSV: ```auto url,status https://attacker-ip.com/1234,online http://malicious-address.br/abc,online ``` Is there a way to search through the data adapter by using a regex like e.g. ._malicious-address.br._ ---
**Author:** ![ihe](https://avatars.discourse-cdn.com/v4/letter/i/a88e57/32.png) [@ihe](https://community.graylog.org/u/ihe)\ **Post date:** [October 10, 2022, 12:38pm UTC](https://community.graylog.org/t/search-through-csv-data-adapter-via-regex/26088/2 "2022-10-10T12:38:17Z")
just to recap you problem: you have a CSV with ```auto url,status https://attacker-ip.com/1234,online http://malicious-address.br/abc,online ``` and you have a field in one of you messages with ```auto .malicious-address.br. ``` now you want to to a little regex-magic to find that domain in your CSV? I would recommend to extract all the domains from your CSV into another CSV and search in that from your lookup table. Then extract the exact domain name from your field and use this for the search in the new domain-only CSV. ---
**Author:** ![dmuensterer](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/dmuensterer/32/13602_2.png) [@dmuensterer](https://community.graylog.org/u/dmuensterer)\ **Post date:** [October 10, 2022, 2:32pm UTC](https://community.graylog.org/t/search-through-csv-data-adapter-via-regex/26088/3 "2022-10-10T14:32:27Z")
That’s exactly what I want to do. Okay, seems like it’s not possible then, unfortunate! ---
**Author:** ![gsmith](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/gsmith/32/1222_2.png) [@gsmith](https://community.graylog.org/u/gsmith)\ **Post date:** [October 10, 2022, 9:50pm UTC](https://community.graylog.org/t/search-through-csv-data-adapter-via-regex/26088/4 "2022-10-10T21:50:20Z")
Hello, If you can give use some more info or examples, perhaps I can lab. ---
**Author:** ![dmuensterer](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/dmuensterer/32/13602_2.png) [@dmuensterer](https://community.graylog.org/u/dmuensterer)\ **Post date:** [October 11, 2022, 7:54am UTC](https://community.graylog.org/t/search-through-csv-data-adapter-via-regex/26088/5 "2022-10-11T07:54:38Z")
Hi gsmith, thanks for the reply. Please have a look at the Github issue where I’ve explained it with some example more in detail: > > > \## What? > > I would be great to have a feature to use lookup tables in search qu…eries. > I.e. for every event matching the search condition so far, make a lookup in the lookup table. > > Useful queries to be supported could be: > \- equals > \- matches (regex matching) > \- contains > \- startswith > \- endswith > > For a given lookup Table \_ioc-lookup\_ and a key \_domain\_, a search syntax could look like the following: > \`\_exists\_:domain AND lookup(ioc-lookup, equals(domain), destinationField)\` > This should only return results if the field \_domain\_ exists and if there is an entry for the respective domain field in the lookup table and set the field destinationField to the looked up value for those results. > > another example could be: > > \`\_exists\_:url AND lookup(ioc-lookup, matches(url, ".\*(?:.com|.net).\*"), destinationField)\` > This should only return results where the field \`url\` exists and matches the regex, i.e. .com and .net domains with any string before and agter. > The search could return after the first match if there were multiple matches found in the lookup table and set the field destinationField to the looked up value for those results. > > \## Why? > > > Let’s assume all client DNS lookups are logged. A user clicks on a phishing link and enters his credentials. > Since the phishing site is very new, the domain is not contained in any IOC database and therefore a lookup of the domain in the Pipeline is not successful. > 10 days later, the phishing domain is known and in the lookup table. > It would be awesome to be able to make a search if any client was affected in the past. > > > > \## Your Environment > Applicable for any environment. I’m looking forward to hearing your feedback on feasibility. ---
**Author:** ![ihe](https://avatars.discourse-cdn.com/v4/letter/i/a88e57/32.png) [@ihe](https://community.graylog.org/u/ihe)\ **Post date:** [October 11, 2022, 9:38am UTC](https://community.graylog.org/t/search-through-csv-data-adapter-via-regex/26088/6 "2022-10-11T09:38:48Z")
Quote from the git: > Why? > Let’s assume all client DNS lookups are logged. A user clicks on a phishing link and enters his credentials. > Since the phishing site is very new, the domain is not contained in any IOC database and therefore a lookup of the domain in the Pipeline is not successful. > 10 days later, the phishing domain is known and in the lookup table. > It would be awesome to be able to make a search if any client was affected in the past. I agree, that searching through your older logs for new IOCs is a valid scenario. But I don’t see a way to put old logs through the pipeline once again. [Decorators](https://docs.graylog.org/docs/decorators-plugins) will not do, as they only process logs shown on your screen and it is not possible to filter based on results of their decorations. So far we are helping ourselves with a litte script doing queries on the stream. Everytime we add a new IOC we scan the last days for it. This is a little python-magic outside of Graylog, but does the job. If we have findings we send them as new logs into Graylog with GELF from that script. ---
**Author:** ![dmuensterer](https://sea2.discourse-cdn.com/flex016/user_avatar/community.graylog.org/dmuensterer/32/13602_2.png) [@dmuensterer](https://community.graylog.org/u/dmuensterer)\ **Post date:** [October 11, 2022, 10:08am UTC](https://community.graylog.org/t/search-through-csv-data-adapter-via-regex/26088/7 "2022-10-11T10:08:54Z")
Exactly, I don’t see the solution in putting old logs through the pipeline again but much rather implementing a feature similar to your Python solution into Graylog, that is, being able to use lookup tables not only for decorators and pipelines but also search queries. This feature would allow endless opportunities for log enrichment and searching. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex016/uploads/graylog/original/3X/c/7/c7c09c6b5099570133d6502b83f50ba4430de5b6.png) [@system](https://community.graylog.org/u/system)\ **Post date:** [October 25, 2022, 10:09am UTC](https://community.graylog.org/t/search-through-csv-data-adapter-via-regex/26088/8 "2022-10-25T10:09:07Z")
Closed on October 25, 2022, 10:09am UTC This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.