Remove date from message

I have some logs that send timestamps in their messages, which I cannot change.
Hence I want to create a rule to catch and remove that. I have tried the following - unsuccessful so far (I have tried with “true” instead of the contains rule, too):

rule "Remvove date"
when
    contains(to_string($message.application_name), "mytimeapp", true)
then
    regex_replace(".*([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}).*", to_string($message.message), "", false);
end

Hello && Welcome @Chris_11

By chance is this a issue? If this is… might want to change category to Graylog Central, This is basically for sharing Ideas and templates members created to help other out. :+1:

Can you post a example of the message?

Hi @gsmith and thank you!
I’m not sure whether this is an issue, or if I’m doing something wrong (newbie here :slight_smile: ). Here’s an example message that should fit (at least regex- wise):

2022-08-10 02:39:18 DEBUG base_http.call_openapi:415 [...someothercontenthere...] [200]

Total understand, I need to find out what your trying to do.
BUT :smiley:
Since you have trouble with pipeline I consider this a incident. I moved it to Graylog Central so others can jump in help ya also.
As for the pipeline, I’m sorry I don’t understand exactly you want.
What I do understand is that you want to remove data from a field? Is this correct?

If so, this data under the fields called application_name? which is the message you just posted?

Hey,

Did a quick forum search on this, maybe this might help.

Thanks you - not really - or, not that I’m aware of :slight_smile:
Ok let’s sum this up: I have a message coming in from a syslog- forwarder. Hence, the “full_message” (which I can see in the graylog search) is:

<190>1 2022-08-10T02:40:25.902071+02:00 my_server mytimeapp - - - 2022-08-10 02:40:25 DEBUG base_http.call_openapi:415  [...someothercontenthere...] [200]

the “message” is shown as:

2022-08-10 02:39:18 DEBUG base_http.call_openapi:415 [...someothercontenthere...] [200]

and the “application_name” is

mytimeapp

Now, I’m confronted with a timestamp from syslog (2022-08-10T02:40:25.902071+02:00), and one from the actual application (2022-08-10 02:39:18). I want to remove the timestamp (plus one whitespace) from the application log and leave only the syslog timestamp.
So my idea was, to filter only messages that I can catch with that specific application name:

contains(to_string($message.application_name), "mytimeapp", true)

and replace the timestamp (regex: ([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}) ) with an empty string (“”):

regex_replace(".*([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}).*", to_string($message.message), "", false);

What am I doing wrong?
Thank you!

Here are a few things I noticed:

  • you can make the when portion more efficient by just checking to see if application_name equals mytimeapp rather than searching for it.

  • You need to assign the results of your regex_replace() back onto the field you are interested in with set_field()

rule "Remove date"
when

    to_string($message.application_name)  == "mytimeapp"
    
then
    set_field("message", regex_replace(".*([0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}).*", to_string($message.message), "", false));
end

That should work for you. :smiley:

1 Like

Thanks a lot, that was a very helpful Input!
I’m almost there, had to make some adjustments:

rule "Remvove duplicate date"
when
    to_string($message.application_name) == "mytimeapp"
then
    let new_message = regex("^\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2} (.*)", to_string($message.message));
    set_field("message", to_string(new_message));
end

Now: the “full_message” looks fine, but the “message” has some sort of encapsulation “{0=…message…}”:

{0=DEBUG base_http.call_openapi:415 [...someothercontenthere...] [200]}

Any ideas how to get rid of that?

I think you can do it like this:

set_field("message", new_message["0"]);

Awesome, that solved it - thank you very much!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.