Ran into a super weird issue that I’m still trying to wrap my head around.
Went to look at a dashboard showing details from one of my nginx instances and noticed the widget that usually shows the Quick values chart for the response_status field received in the last 24hrs was showing N/A with an exclamation point in a red triangle & “Loading…” in the top right.
At first look not so weird, but, the same thing happened to the same widget in every dashboard for all our nginx instances. Additionally attempting to go to the nginx streams themselves and hitting “Quick values” on the response_status field results in a 500 error. This is consistent across all of my nginx streams which comprise output from four entirely separate EC2 instances running nginx.
Even stranger, I was able to narrow it down to a specific time. The dashboards show Quick values from the last 24h, and that’s the query that fails with a 500. Getting quick values for all the relative time periods up to and including 8 hours works fine. The jump to 1 day results in a 500.
By stepping back through logs using the absolute timeframe I was able to pin it down to 2017-05-23 19:00:06. Any attempt to use the “Quick values” function on the response_status field prior to that time on any of the nginx streams results in a 500.
This is super bizarre because this widget has been on these dashboards for three months at this point with no issues. Moreover, the actual response_status fields from the logs prior to that date appear just fine, it’s only the Quick values functionality that’s broken.
Any thoughts on this would be welcomed as I’m really at a loss to even come up with a mechanism by which this could happen.