Best Method for Forwarding A Variety of Application ".Log" Files

Graylog Add-ons
, , , ,
1

Hey Guys -

I’m new to GrayLog and even though considering using it for work, am trying it out at home first. I’ve already added the appliance, am sending Windows Event logs using nxlog, and seem good to go; but have a question about how to configure what I’m wanting to do.

I have a few different apps that I host on my home PC and I’d like to have the log files from all of them go into Gray log so they’ll be in a central location. The thing is, most of them don’t offer anything close to syslog or other forwarding capabilities - they just write to log files.

My Question

So given the above, would be the best way to have these feed into GrayLog? I don’t know if I need to use nxlog agent, a Marketplace plugin / content pack, manual configuration, a combination of them, or something different altogether. Below are some examples of them and their formatting. All are running on Windows

App #1

  • Writes to a .log file which is rolled over each time it gets to 1mb (about every 2 days)
  • Examples of the log file formatting
    06-Apr-2017 01:58:16 - INFO :: Thread-21 : Parsing results from
    06-Apr-2017 01:58:18 - INFO :: Thread-21 : No results found from
    06-Apr-2017 01:58:18 - INFO :: Thread-21 : Parsing results from

App #2

  • Writes to a .txt file which is rolled over when it gets to 1mb
  • Examples of the log file formatting
    17-4-6 10:15:15.6|Error|XbmcMetadata|Unable to process episode image for file
    [v2.0.0.4688] System.InvalidOperationException: Sequence contains no elements
    at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
    at NzbDrone.Core.Extras.Metadata.Consumers.Xbmc.XbmcMetadata.EpisodeImages(Series series, EpisodeFile episodeFile) in M:\BuildAgent\work\b69c1fe19bfc2c38\src\NzbDrone.Core\Extras\Metadata\Consumers\Xbmc\XbmcMetadata.cs:line 345
    17-4-6 10:15:16.1|Info|RefreshEpisodeService|Starting episode info refresh
    17-4-6 10:15:16.2|Info|RefreshEpisodeService|Finished episode refresh
    17-4-6 10:15:16.2|Info|DiskScanService|Scanning disk
    Log file also seems to include some HTML code

App #3
Simple - Apache 2.4 for Windows

App #4

  • Writes to a .log file which is rolled over when it gets to 5mb
  • Examples of the log file formatting
    2017-04-02 19:48:28,349::INFO::[postproc:87] Saving postproc queue
    2017-04-02 19:48:28,351::INFO::[init:971] Saving data for postproc2
    2017-04-02 19:48:28,351::INFO::[downloader:300] Post-processing finished

App #5

  • Writes to a .log file which is rolled over when it gets to 500kb
  • Examples of the log file formatting (This one seems to have non-standard characters in it)
    03-21 21:37:51 ERROR e[31mFailed encoding stuff to log “Cant find it in database. Are you sure this it exists?”: not all arguments converted during string formattinge[0m
    03-21 21:37:51 ERROR e[31m[e.providers.com] Cant find it in database. Are you sure this it exists?e[0m
    03-21 21:37:51 INFO e[0m[core.media.movie.searcher] Search for Title ignoring ETAe[0m
    03-21 21:37:52 INFO e[0m[hpotato.core.plugins.base] Opening url: get http://localhost:9117e[0m
    03-30 17:22:18

I’m really only concerned with the first 3 so if steps dramatically change between them; they don’t have to be included. Any suggestions?

Thanks!!!

2

Hej @bzowk

as you already use nxlog you might want to use that to grep the files and deliver them to graylog.

Personal I would use filebeat to get the log files and winlogbeat to get the windows event log - but that is just me, personal.

Did you configure nxlog with the Collector Sidecar or did you configure it direct?

3

Thanks for the reply!

I actually configured it using a conf template I found online somewhere as still trying to learn the basics. It not only had a template, but also instructed me to create a new input for it (which I did.) The input is a GELF UDP which it had me set to all defaults except for enabling “Global.”

I don’t care how things are forwarded - just would like to learn the best way to do it and perhaps why. I’ll check out Filebeat and WinLogBeat in a few and set them if suggested. Hopefully, I can find an article which will describe which inputs need to be created and an overview of the process.

Anything you could point me towards would be great - Thanks again!

4

Hej Ben,

I think you already looked into the Graylog Documentation.