Categorize files in graylog

(Rayees Namathponna) #1

Here is my usecase, I have only one input and logs harvesting from different application lets take tomcat, tomcat have files like catalina.out, access.log, tomcat.out etc, i want to group all these files into one, example SOURCE_TYPE=tomcat

I should be able to search all above there files with this string SOURCE_TYPE=tomcat , any suggestion ???



The method I use is to get nxlog to add a new field called (for me ‘sourcetype’).
This is configured in nxlog input configuration for a collector.
For example for my IIS logs …

Exec $size = size($raw_event);
Exec if $raw_event =~ /^#/ drop();
$EventTime = parsedate($date + " " + $time);
$EventTime = strftime($EventTime, “%Y-%m-%dT%H:%M:%SZ”);
$sourcetype = “fred”;

Then you can search for ‘sourcetype=fred’

(Rayees Namathponna) #3

Thanks @harryw, I am using filebeat , any idea how to make this with filebeat

(Jan Doberstein) #4

when you are using filebeat you have a field that includes the source of logfile. This is in the field ‘file’ and you can create a pipeline that generate what you like.

(Rayees Namathponna) #5

In beats input confiuration, we can add fields and value in key value pair , its getting added to each message, I think we can use this itself without using pipeline (BTW I didnt understood how to use pipeline in this usecase), is there disadvantage by using this way, I know, I need to configure multiple beats inputs in this case, anything else ???