Issues with date conversion in pipeline

#1

Hello,

I’ve been trying to write a pipeline function that takes the end time of a VPN session (this is essentially pulled from the timestamp), the duration of the session, and then computes the start time of the session from these two values. However, I’ve tried numerous ways of implementing this to no avail.

The underlying idea was that I could convert the human readable date from the timestamp into UNIX time, subtract the session duration (in seconds), and then convert the result back into a human-readable date as the start time field. However, I’ve been unable to find a way to reliably convert the human readable date in the timestamp into seconds.

Any assistance would be greatly appreciated. Thank you in advance.

0 Likes

(Jan Doberstein) #2

what did you have currently? How did you tried? What does the rule look like?

That is something I have shown on a presentation:

https://cdn.rawgit.com/jalogisch/OpenSourceDay2018/d3ffdebf/Presentation.html#32

0 Likes

(Ben van Staveren) #3

Hunh, I didn’t know that was possible. Does parse_date (and flex_parse_date I guess) return a Date object? I haven’t been able to find much about that in the docs :confused:

0 Likes

(Jan Doberstein) #4

it does - and @benvanstaveren, I know that we need to improve that. But so much work so little time.

0 Likes

(Ben van Staveren) #5

No worries @jan - been there, done that, still waiting on the t-shirt :smiley: I wish I had more time myself, I’d try and assist but it’ll take me a while (with my current amount of free time) to understand it all, and then some more time to write docs. I’ll give it a shot, though! :smiley:

0 Likes

#6

Jan,

My apologies for the delay in responding, as work was extremely hectic yesterday. I reviewed the information in the link provided and tried to do something similar to what you had (only I’m working with a timestamp and a duration to calculate a date/time value vs. working with two timestamps to calculate a duration).

Initially, after reviewing what you had shared, I had tested something along the lines of the following:


  let VPNSessionDurationSec = ((durationDays * 86400.00) + (durationHours * 3600.00) + (durationMinutes * 60.00) + durationSeconds);

  let VPNSessionStartTime = (parse_date(value: to_string(VPNSessionEndTime), pattern: "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'").seconds - VPNSessionDurationSec.seconds);

  set_field("VPNSessionStartTime", VPNSessionStartTime);

As a note, the “VPNSessionDurationSec” variable is already in seconds, but without the “.seconds” at the end, Graylog gave an error that the two variables were two different types or something along those lines. After deploying this updated code, I received the following in one of the new log messages:


gl2_processing_error

For rule 'convert session duration and calculate session start time': At 21:29 an exception was thrown: null

Line 21 is the “let VPNSessionStartTime” line, and column 29 is the beginning of the “parse_date” function.

From here, I decided to test and just ensure that the initial date conversion to seconds was working properly, so I commented out the other lines and added the following:


set_field("EndTimeSeconds", parse_date(value: to_string(VPNSessionEndTime), pattern: "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'").seconds);

After testing again, this did not throw an error, but the new message did not have an “EndTimeSeconds” field at all (I tested this twice just to ensure it wasn’t a fluke), so I’m not sure if the conversion is failing or if there’s some other issue here. With that being said, I know that the “VPNSessionEndTime” variable is not null and is in the format specified, and that the “VPNSessionDurationSec” variable is not null and holds the correct data (excerpt from one of the messages below):


VPNSessionEndTime
2019-02-02T01:16:43.000Z

VPNSessionDurationSec
150

In your example (which I attempted to base the updated pipeline processing code off of), you’re converting the dates to strings (via to_string) and then subtracting the REQUESTTIME from the RESPONSETIME (where as I’m trying to subtract the session duration (in seconds) from the end time).
As a quick question, is this correct, given that we’re trying to perform an arithmetic function on two strings vs. converting/casting these variables to numerical values (i.e. – via “to_long” or “to_double”), or does appending the “.[millis|seconds]” to the parse_date() function automatically convert these variables into a numerical-type variable?

Any further assistance would be greatly appreciated, as I’ve been spinning my wheels on this for awhile. Thank you in advance.

0 Likes

#7

Just wanted to bump this to see if there was any further input or advice. @jan, would you per chance be able to provide any further input on the additional details that I provided, or is there further information that is needed? Thank you again for your time and assistance with this.

0 Likes

(Jan Doberstein) #8

sorry currently I do not have the time for that kind of play with data.

0 Likes

#9

@jan, I completely understand. I greatly appreciate the help that you’ve already provided, as it at least gives me another good example to review, as I haven’t worked that extensively with pipeline processing functions in Graylog yet.

0 Likes

#10

Anybody else have any recommendations per chance? Thank you in advance.

0 Likes

(system) closed #11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

0 Likes