I have some logs that send timestamps in their messages, which I cannot change.
Hence I want to create a rule to catch and remove that. I have tried the following - unsuccessful so far (I have tried with “true” instead of the contains rule, too):
rule "Remvove date"
when
contains(to_string($message.application_name), "mytimeapp", true)
then
regex_replace(".*([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}).*", to_string($message.message), "", false);
end
Hello && Welcome @Chris_11
By chance is this a issue? If this is… might want to change category to Graylog Central, This is basically for sharing Ideas and templates members created to help other out. 
Can you post a example of the message?
Hi @gsmith and thank you!
I’m not sure whether this is an issue, or if I’m doing something wrong (newbie here 
). Here’s an example message that should fit (at least regex- wise):
2022-08-10 02:39:18 DEBUG base_http.call_openapi:415 [...someothercontenthere...] [200]
Total understand, I need to find out what your trying to do.
BUT 
Since you have trouble with pipeline I consider this a incident. I moved it to Graylog Central so others can jump in help ya also.
As for the pipeline, I’m sorry I don’t understand exactly you want.
What I do understand is that you want to remove data from a field? Is this correct?
If so, this data under the fields called application_name? which is the message you just posted?
Hey,
Did a quick forum search on this, maybe this might help.
Thanks you - not really - or, not that I’m aware of
Ok let’s sum this up: I have a message coming in from a syslog- forwarder. Hence, the “full_message” (which I can see in the graylog search) is:
<190>1 2022-08-10T02:40:25.902071+02:00 my_server mytimeapp - - - 2022-08-10 02:40:25 DEBUG base_http.call_openapi:415 [...someothercontenthere...] [200]
the “message” is shown as:
2022-08-10 02:39:18 DEBUG base_http.call_openapi:415 [...someothercontenthere...] [200]
and the “application_name” is
mytimeapp
Now, I’m confronted with a timestamp from syslog (2022-08-10T02:40:25.902071+02:00), and one from the actual application (2022-08-10 02:39:18). I want to remove the timestamp (plus one whitespace) from the application log and leave only the syslog timestamp.
So my idea was, to filter only messages that I can catch with that specific application name:
contains(to_string($message.application_name), "mytimeapp", true)
and replace the timestamp (regex: ([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}) ) with an empty string (“”):
regex_replace(".*([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}).*", to_string($message.message), "", false);
What am I doing wrong?
Thank you!
Here are a few things I noticed:
you can make the when portion more efficient by just checking to see if application_name equals mytimeapp rather than searching for it.
You need to assign the results of your regex_replace() back onto the field you are interested in with set_field()
rule "Remove date"
when
to_string($message.application_name) == "mytimeapp"
then
set_field("message", regex_replace(".*([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}).*", to_string($message.message), "", false));
end
That should work for you.
Thanks a lot, that was a very helpful Input!
I’m almost there, had to make some adjustments:
rule "Remvove duplicate date"
when
to_string($message.application_name) == "mytimeapp"
then
let new_message = regex("^\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2} (.*)", to_string($message.message));
set_field("message", to_string(new_message));
end
Now: the “full_message” looks fine, but the “message” has some sort of encapsulation “{0=…message…}”:
{0=DEBUG base_http.call_openapi:415 [...someothercontenthere...] [200]}
Any ideas how to get rid of that?
I think you can do it like this:
set_field("message", new_message["0"]);
Awesome, that solved it - thank you very much!
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.