Ingesting CSV files

Graylog Central (peer support)
1

Hi,

We’re attempting to ingest CSV files from a few different sources, but as an example generally they are structured as below:

ID,AuditIndex,Authentication,CSPSessionID,ClientExecutableName
User1,1865062,Password,WjqwJMukW2,CSPap.so
User2,1865063,Password,RfkqkMruqM2,CSPap.so

In attempting to write pipeline rules to properly break up and label all these values, the issue I’m running into is that there doesn’t seem to be any functionality that allows me to loop through all lines ingested to apply these rules. Using a lookup table to temporarily store and process won’t work for me either, as the CSV I’m ingesting uses no quote characters and the CSV data adapter doesn’t seem to like this.

Is there any way to loop through pipeline rules in the current graylog version that I’m not seeing? Or am I approaching this the wrong way entirely?

Thank you-

2

You would send each line from the CSV file to Graylog (for example with Filebeat) and can process each message with a Grok pattern matching the columns of your CSV file.

3

If I were to simply echo the contents of the file line-by-line and send via nc, would that accomplish the same thing more or less?

4

Yes, as long as you’ve started an input receiving this data in Graylog (e. g. Raw/Plaintext TCP or UDP).

5

Hi again,

My apologies for my ignorance and hopefully it’s okay I’m responding a week after the thread was opened - I can’t seem to get a Grok pattern figured out for the CSV I’m ingesting. I can’t seem to find any samples or anyone else needing the same things on the forum, either. I feel like I must be looking at this the wrong way.

Example of the data formatting-

ID,AuditIndex,AuthType
1865062,904456,Password

6

What did you try so far and what was the result?

7

Hi,

I wrote a small grok pattern just as a sort of test to see if it would parse out the first field of my data, and it does - however upon trying to save the pattern I get this error kicked back from Graylog:

Could not save Grok pattern
Saving Grok pattern “CSV test 1” failed with status: cannot POST http://10.10.50.30:9000/api/system/grok (400)

I assumed this was a problem with my pattern but is there maybe a larger issue with my Graylog install?

8

Check the logs of your Graylog nodes.
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

9

Thanks man, will do. Appreciate it

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.