We are running into an issue with several of our graylog instances where the search page does not load correctly. It will add about 8 to 14 mins of time ahead of the latest event. When we load the search page and it uses the default search of last 5 mins, it will not show anything because it’s actually looking at the last 5 minutes that are about 12 minutes in the future.
in addition, if we remove the last 5 mins search from configuration and change it to last 15 mins - when clicking the search button at the top it will show nothing found. We have to then click the green magnifying glass within the search page to load the search. Once again it is adding the extra time to the search page
This is happening with several graylog instances and there is nothing in common across all instances.
Most are running 2.2.3 with elasticsearch 2.4.4. Some were running 2.1.1 and upgraded to 2.2.3 and it did not make a difference. We have not upgraded to 2.3 on any instances yet.
possibly related, but that issue would be a full day’s of events not showing due to time zone differences, our issue is an extra 8 to 14 mins that is added to the search screen. We have tried applying an rsyslog template to add %timegenerated% to each syslog message when it hits the syslog collector before it is forwarded over to graylog. graylog parses these events properly and this eliminates an possibility of events being slightly in the future or past due to drifting timestamps from networking devices.
any other ideas? i’m willing to rebuild these graylog instances from scratch again if needed
Any ideas? my temp fix would be to have a device send events in the future at a timed interval so that the histogram is populated with something and the page doesn’t display with nothing. This is happening to at least 4 client platforms. 2.3.1 does not correct the issue. going to attempt to upgrade java next
Figured it out and it was something very simple that we overlooked. the system time was wrong. we assumed everything was in sync because the overview page showed the correct time. So it appears that the histogram honors the system time, but everything else goes by the browser time and nothing looks at the hcclock time. I’ll be filling out a bug report about the time not reflecting correctly on the overview page, thanks
