Configuration: Tailor the inputs or the extractors and Syslog default formatting

[texas-bronius (Bronius Motekaitis)]

First time syslog forwarder, long time logger Graylog looks absolutely terrific, and the most recent release has all kinds of useful, new capabilities!

Question 1: Configure before syslog or extract after?

Is there a rule of thumb that could be said: Let the application spit out whatever log and format it wants out of the box, and let the log analysis tool parse and chunk it per-application? This, as opposed to excessive customization of each application’s log to the extent to which we are given that control and with the likelihood that still some custom extractors will be required.

Question 2: Default time has no year-- assume this year?

And with syslog: On Centos, rsyslog by default puts a timestamp -of the log message- as:
Oct 20 13:54:33
There is no year in it…? From my reading on Extractors, I gather Graylog converts this timestamp into a date in this current year. Is this correct?

Question 3: Importance of Log timestamp vs Event timestamp

How important is the syslog timestamp when we are most likely to include an event timestamp in our log files themselves? Does much of Graylog depend on this timestamp as being assumed to be about the same time as the event, or?..

Thanks!
-Bronius

[jochen]

@texas-bronius

Is there a rule of thumb that could be said: Let the application spit out whatever log and format it wants out of the box, and let the log analysis tool parse and chunk it per-application? This, as opposed to excessive customization of each application’s log to the extent to which we are given that control and with the likelihood that still some custom extractors will be required.

The more preprocessing your applications can do the better. Ideally use JSON or GELF and structured logging.
Check the Graylog Marketplace for existing GELF appenders for your logging frameworks:

the NEW Marketplace

@texas-bronius

There is no year in it…? From my reading on Extractors, I gather Graylog converts this timestamp into a date in this current year. Is this correct?

Yes, but that’s bad practice. Use the proper syslog message template in rsyslog and you’ll be much happier (and the syslog messages will contain a proper timestamp):

Graylog2/graylog-guide-syslog-linux/blob/master/README.md#rsyslog

@texas-bronius

How important is the syslog timestamp when we are most likely to include an event timestamp in our log files themselves? Does much of Graylog depend on this timestamp as being assumed to be about the same time as the event, or?

There’s a difference in the message timestamp and the time the message was received by Graylog. You usually want to have a proper (and correct) message timestamp, ideally ISO 8601 with date and time in UTC.

[texas-bronius (Bronius Motekaitis)]

Excellent response on all points, thanks.

In particular, I am surprised by the recommendation to try to train each application, especially considering this prevents any historical log import and analysis. And yes, I am kicking myself for not having instituted json years ago! Hmm… I suppose it’s possible to create a separate log entry just for syslog just in json in the case of apache.

The other two points (and with specific instructions cited!) I can run with to the bank without qualm. Cheers!
-Bronius