My log format is such that the time stamp is not recognized, as such the (bold) timestamp shows up as the time the message was ingested rather than the actual time. I am parsing out the actual date/time with an extractor and can see it in my search results, I’m struggling however to override the “Timestamp”.
I’m trying to follow this Set timestamp with pipeline solution but I’m not entirely sure I get the process. I created a pipeline which has a Stage 0 to which my rule is attached, I guess I don’t entirely understand why this is attached to a “Stream”.
Right now “Timestamp” isn’t getting overridden, I added a second new field to my rule to see if it is created and it is not, as such I suspect maybe the rule is correct (I copied it from the link above) but that I don’t have it hooked in correctly.
Is this the right procedure… create a Pipeline, Create a Rule, Add a Stage 0 to the Pipleline and specify the rule, attach the Pipeline to the “All Messages” Stream?
I have an Extractor that parses this out into a field called myDateStamp, I think I have just verified that my rule will run (or not) suggesting I have syntax issues, for example if my rule is this:
rule “parse event timestamp”
when
true
then
set_field(“foo”,“Bar”);
end
Then I see the field “foo” for every message.
However if I make the rule this:
rule “parse event timestamp”
when
true
then
set_field(“foo”,“Bar”);
let new_date = parse_date(to_string($message.myDateStamp), “yyyy-MM-dd’T’HH:mm:ss.SSSZ”);
set_field(“timestamp”, new_date);
end
I suspect the rule is failing as I don’t get either field populated.
Based on various combinations of tests it appears that the rule breaks whenever I reference the extracted field myDateStamp, I’m creating this using an extractor, I can see it in my messages just fine but if I put it in a has_field(“myDateStamp”) or I reference it in a rule it seems the whole rule breaks.
Is there a limitation on using fields created in an extractor?
Do I need to prefix it with $message or something maybe?
