Here is my usecase, I have only one input and logs harvesting from different application lets take tomcat, tomcat have files like catalina.out, access.log, tomcat.out etc, i want to group all these files into one, example SOURCE_TYPE=tomcat
I should be able to search all above there files with this string SOURCE_TYPE=tomcat , any suggestion ???
HI.
The method I use is to get nxlog to add a new field called (for me ‘sourcetype’).
This is configured in nxlog input configuration for a collector.
For example for my IIS logs …
Exec $size = size($raw_event);
Exec if $raw_event =~ /^#/ drop();
else
{
w3c->parse_csv();
$EventTime = parsedate($date + " " + $time);
$EventTime = strftime($EventTime, “%Y-%m-%dT%H:%M:%SZ”);
$sourcetype = “fred”;
}
Then you can search for ‘sourcetype=fred’
when you are using filebeat you have a field that includes the source of logfile. This is in the field ‘file’ and you can create a pipeline that generate what you like.
In beats input confiuration, we can add fields and value in key value pair , its getting added to each message, I think we can use this itself without using pipeline (BTW I didnt understood how to use pipeline in this usecase), is there disadvantage by using this way, I know, I need to configure multiple beats inputs in this case, anything else ???
