Histogram Timestamp Resolution Problems

Hi all,

Apologies in advance for external images, the forum will only let me post one and link to two.

I’m having a curious time display problem that I just can’t get to the bottom off.

Currently on Graylog 3.0.3.

The timestamps on overview screen match my current workstation time:
image

I am ingesting logs from logstash via gelf and have an accurate count from bash of log lines:

cat *.log* | egrep "\[type_name:10:deliver_sm\]" | wc -l

214741

This matches my count in graylog and I am happy:

If I pick on a single day in bash I get 4275:

cat .log | egrep “[type_name:10:deliver_sm]” | cut -d’ ’ -f1 | sort | uniq -c | grep 2020-07-29
4275 2020-07-29

Here is where it gets interesting.

Depending on HOW i search it in graylog I may not get matching figures:

Wrong (histogram for same day shows wrong count):

The same thing happens if I type in “Yesterday” using relative.

If however I use absolute time:
hxxps://i.imgur.com/sFR4XyW.png

I get the correct total amount but the day histogram shows a split of the correct total on 28th and 29th?

If I change resolution to hour, the 28th disappears (first bar appears as 29th upon hover):
hxxps://i.imgur.com/CbQSWVJ.png

The earliest record displayed in results is the 29th (correct), not the 28th:
hxxps://i.imgur.com/MSLvlm4.png

What could explain this behavior? I went through a very long exercise of making sure system timezone, usertimezone, root_timezone, ntp, etc we’re all in sync and graylog seems to be agreeing with me.

Am I missing something?

Would really appreciate some input. Racking my head for a week over this.

Edit: formatting

Went for a third google-fu session and found this. Is this what I’m dealing with?

Which lead me to:

Which is still an open issue.