Pipeline rule to convert timestamp field and parse message json

Graylog Central (peer support)
1

Hello everyone!
Previously I encountered an error: gl2_processing_error (Replaced invalid timestamp value in message)
I solved this by converting the date. But if leave this one condition with the conversion, then the message will not be parsed (usually I used default json put there us a gl2_processing_error)

I added lines to parse the json message and when I run the simulation the rules are fine. All fields are laid out. But when I go to the stream and open any log, I just have only a whole message and a timestamp field there
Here is my rule:

`rule "replace timestamp"
when
    true
then
    let result = regex("([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{9}Z)", to_string($message.message));
    let new_date = parse_date(to_string(result["0"]), "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSS'Z'","UTC");
    set_field("timestamp", new_date);
    let json_parsed = parse_json(to_string($message.message));
    set_fields(to_map(json_parsed));
end`
2

So ever so often the way pipelines see data is different than how it ends up being stored in opensearch. You might want to try the debug function and look in server.log to just make sure its proper json when it hits the pipeline.

Also i tend to prefer flatten_json as it seems to give me less challenges.

3

I`ve tried to use flatten… nothing changed…

    let json_parsed = flatten_json(to_string($message.message), "flatten");
    set_fields(to_map(json_parsed));
    set_field("message", "parsed user data");

About server.log, I don’t have the ability to view from the server side… Need to push devops, I wish I didn’t have to do this

4

If you have 5.2 you can view the most recent log messages under system>nodes>more actions view logs

5

Oh thx! Yep, I see some errors…

[PipelineInterpreter] Error evaluating action for rule <replace timestamp/null> in pipeline <dummyPipeline/stage 0> with message: source: null 

(Error: In call to function 'parse_date' at 6:19 an exception was thrown: Invalid format: "")
6

what part of message has an Invalid format… empty quotes - what is this?

7

and other errors:

Rule <replace timestamp> Unable to parse JSON
com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'INFO': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')

I`ve rewrited my rule:

rule "replace timestamp"
when
    true
then
    let result = regex("([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{9}Z)", to_string($message.timestamp));
    let new_date = parse_date(to_string(result["0"]), "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSS'Z'","UTC");
    set_field("timestamp", new_date);
end

And try to only parse message:

 rule "message parser"
when
    true
then
    let json_parsed = parse_json(to_string($message.message));
    set_fields(to_map(json_parsed));
end

Nothing changed… I really need help=)

8

Can you post an example message, its okay if you need to obviscate some fields as long as its similar enough that the same regex works

9

Not sure if it will help, but here is what I used to parse json. Extract nested json issue - #3 by rfinney

10

sure, Im on a vacation, as I get back to work, Ill show my message.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.