Just tried to start sending netflow data from my Untangle 16.4.1 to my Graylog server (v4.0.14, on Debian 11) using Netflow V9, but all the incoming messages started at timestamp Dec 31, 1969, and have been slowing incrementing.
Heres some pictures:
I don’t even know what is going on here, or if there is any way to solve this?
Has anyone seen this? How do I fix this?
Under System->Overview, scroll down to Time configuration - double check there. What Input are you using? There is a Netflow UDP input… If it’s not giving you the right information, you may need to switch to a RAW input.
Time configuration
User admin: 2022-01-20 17:40:37 -05:00
Your web browser: 2022-01-20 17:40:37 -05:00
Graylog server: 2022-01-20 17:40:37 -05:00
I am using the UDP Netflow input, I will try and switch to raw.
I don’t think UDP raw is going to be a better solution. The timestamps are right, but all the messages are truely indeed RAW:
** h2:�mT��a0�0� M�’B�a�0�0��’B�e��0�0�=�>'B��0�0��’B�>��0�0�=�’B��0�0��’B���0�0�**
Hmmm - Perhaps settings on the sending side? I haven’t worked with Netflow but wanted to get the thread started…
Untangle has very few configuration options, just the host, port, and netflow version.
Hey @dscryber anyone you can check with for Netflow Input questions - this has an odd timestamp issue…
Sure. I’ll check it out.
BTW: Tadd, I’d like to highlight you in the newsletter. Would you please send me three or four sentences about you (like a short bio) and say one thing about your CAB appointment in the community?
Also, please send me a picture you’d like to include in the newsletter. I’m scheduled to send out the newsletter for Jan 31.
Thanks!
–David
Hello,
I have been using Netflow for quit a while,
Double check your server ( NTP) and Untangle 16.4.1 Date/time.
I found these links.
https://wiki.untangle.com/index.php/Time_and_date_formatting
Also since you seam to be dealing with 1/1/1970 the "epoch time"
I have found this post
EDIT: With Untangle 16.4.1 I think you can change the date, time, and year from a prompt with this command.
date 060222212009 translate to January 21, 10:21pm year 2020
Other than that, the date command on the console is the only way.
Hope that helps
Checking out Untangle from SSH, everything seems fine with the clock:
# timedatectl
Local time: Fri 2022-01-21 22:04:26 EST
Universal time: Sat 2022-01-22 03:04:26 UTC
RTC time: Sat 2022-01-22 03:04:19
Time zone: America/New_York (EST, -0500)
System clock synchronized: yes
NTP service: inactive
RTC in local TZ: no
I will say, I don’t have my firewall connected to command center. But I don’t know why that would affect netflow?
Hello,
Untangle seams to be sending epoch time That’s the reason I gave you those links to check out
other community members in Untangle forum that seam to have the same problem. Elasticsearch received those messages as epoch time.
If your 100% sure that there is not a problem on Untangle then the next post should show how to adjust it that type of Date/time if you cant adjusted in Untangle.
EDIT : Sorry My epoch time is off in the example above, you may want to look here if you need to adjust Date/Time
I’m a little lost by what you are saying.
I’m not sure what problem could be occuring with Untangle, but all I can assume is that this is a graylog issue, as other logs (From untangle) do not have this timestamp problem. Just netflow.
If the problem is how Untangle is sending netflow, then that is out of my control.
The messages being received by graylog have the timestamp in this format: “1970-01-22 23:57:13.306 -05:00”. They do not arrive in seconds since 1970. They might arrive in seconds and graylog is formatting it, but that is out of my control.
If the other untangle devices does not have Netflow enable, then I assume there send syslogs. Then the timestamp would be correct.
If the untangle device that has Netflow enable on it sending logs to Graylog using INPUT Netflowv9 it now sending those message in epoch time. In my environment I had to adjust the Date/Time after enabling Netflow. The links above stated that the only way to adjust this in in the console of the Untangle device.
Are you able to capture those message being received from the untangle device with Netflow enabled before they are ingested through Graylog? Then you will know for sure if it Input or the remote device has the incorrect date/time.
Hey all,
Couple of quick points, I don’t mind blaming Graylog but a lot of folks are using the NetFlow input swimmingly, so let’s see what else we can do.
Double check you’re Netflow version on Untangle is v9
Try just using your Graylog server as the raw receiver. This will grab the raw data on port 2055 UDP and dump it to a file called netflow_capture. Modify as you need and point Untangle to send to the port you choose (2055 in this example). Show us that file and we can probably get you sorted.
nc -ul 2055 > netflow_capture
My theory - Untangle is using IPFIX headers, so the epoch theory was probably right. Graylog is seeing a number like 2345 (millisecond since capture) when it expects 1642411499 (ms since 01 Jan 1970, standard epoch time). You’re looking at the converted time stamp in the UI (your screenshots).
NetFlow also has a different time mechanism than syslog, so it’s perfectly logical that one type can work while the other doesn’t.
Without seeing the raw data from Untangled one way or another we can’t point the finger in any direction.
Untangle netflow is on v9.
I have the capture file, can I message it to you? It does contain public IPs.
I also did a packet capture.
Just thinking ahead here, but how would I go about changing every timestamp for the netflow coming into graylog?
Just change them with a find/replace or sed and paste here.
As for how to change, the easy way will be to capture the current time in epoch and add it. That’s the nice part about epoch - it’s just a big integer, so it’s easy to manipulate.
Id like to do that, but I’m gonna need to decode the raw network traffic first…
��������5�E�=�����5a��E�=����@a�5�E�=�����5a��E�=����@a�5�E�=�����5j�E�=����@j5����]5�(�E�=�e,���������*E�=���)� UW���)N
Extractors are not my strong suite, I don’t see any built in option in grok for date in epoch. I assume math can be done in grok extractors?
@erasedhammer
Sorry for the delay, without tagging me I forget.
As for this issue, adding on to what @BlueTeamNinja stated. I think a pipeline would be the way to go.
Here is an example.
Perhaps give that a try and if you stuck just post what you have done here , I sure someone here will respond.