Setting Timestamp - Piplelines/Rules


#1

My log format is such that the time stamp is not recognized, as such the (bold) timestamp shows up as the time the message was ingested rather than the actual time. I am parsing out the actual date/time with an extractor and can see it in my search results, I’m struggling however to override the “Timestamp”.

I’m trying to follow this Set timestamp with pipeline solution but I’m not entirely sure I get the process. I created a pipeline which has a Stage 0 to which my rule is attached, I guess I don’t entirely understand why this is attached to a “Stream”.

Right now “Timestamp” isn’t getting overridden, I added a second new field to my rule to see if it is created and it is not, as such I suspect maybe the rule is correct (I copied it from the link above) but that I don’t have it hooked in correctly.

Is this the right procedure… create a Pipeline, Create a Rule, Add a Stage 0 to the Pipleline and specify the rule, attach the Pipeline to the “All Messages” Stream?


(Brad Booth) #2

Can you post what the actual time appears like (the one you extracted) in your log as well as the rule you have?


#3

Each log line begins with a timestamp like this:

[8/16/15 19:35:50:162 EDT] 0000004a SystemOut

I have an Extractor that parses this out into a field called myDateStamp, I think I have just verified that my rule will run (or not) suggesting I have syntax issues, for example if my rule is this:

rule "parse event timestamp"
when
true
then
set_field(“foo”,“Bar”);
end
Then I see the field “foo” for every message.

However if I make the rule this:

rule "parse event timestamp"
when
true
then
set_field(“foo”,“Bar”);
let new_date = parse_date(to_string($message.myDateStamp), “yyyy-MM-dd’T’HH:mm:ss.SSSZ”);
set_field(“timestamp”, new_date);
end

I suspect the rule is failing as I don’t get either field populated.


#4

I should have said myDateStamp (the extracted field) shows up in search results as, for example, “2015-08-17T13:00:51.000Z”


#5

Based on various combinations of tests it appears that the rule breaks whenever I reference the extracted field myDateStamp, I’m creating this using an extractor, I can see it in my messages just fine but if I put it in a has_field(“myDateStamp”) or I reference it in a rule it seems the whole rule breaks.

Is there a limitation on using fields created in an extractor?

Do I need to prefix it with $message or something maybe?


#6

Okay, after some searching I believe I have resolved the issue with this post https://github.com/Graylog2/graylog-plugin-pipeline-processor/issues/163 and the documented solution where the sequence of processors affects the availability of fields created with extractors.


(system) #7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.