I like to parse and set the correct timestamp to logmessages with the pipeline feature. I have all ready fields with hour,minutes,seconds, day of month and so on…
My first pipeline stage is detecting the logs (for example application server type a). The second stage is parsing the logmessage with the proper grok pattern.
Now I have the issue, that the logmessage timestamp and the graylog (elasticsearch) timestamp differ a view seconds. My idea is to use the parsed fields from stage two and set the correct timestamp in stage three.
My idea is a rule like this:
rule "Appserver Parsing - Timestamp" when true then let new_timestamp = parse_date( value: to_string($message.year) + "-" + to_string($message.month) + "-" + to_string($message.day) + " " + to_string($message.hour) + ":" + to_string($message.minute) + ":" + to_string($message.seconds), pattern: "yyyy-MM-dd hh:mm:ss", timezone: to_string("Europe/Berlin") ); set_field("AAA_pipline_timestamp", new_timestamp); // If the timestamp is correct, rename the field end
Does this is the correct way? Is an other way a better solution?
I have many different log formats and think this will be a simple way to get the values in fileds and then set the correct timestamp.