[Solved] 500 errors for quick values for a single field across multiple streams

[rtanay (Ryan Tanay)]

Ran into a super weird issue that I’m still trying to wrap my head around.

Went to look at a dashboard showing details from one of my nginx instances and noticed the widget that usually shows the Quick values chart for the response_status field received in the last 24hrs was showing N/A with an exclamation point in a red triangle & “Loading…” in the top right.

At first look not so weird, but, the same thing happened to the same widget in every dashboard for all our nginx instances. Additionally attempting to go to the nginx streams themselves and hitting “Quick values” on the response_status field results in a 500 error. This is consistent across all of my nginx streams which comprise output from four entirely separate EC2 instances running nginx.

Even stranger, I was able to narrow it down to a specific time. The dashboards show Quick values from the last 24h, and that’s the query that fails with a 500. Getting quick values for all the relative time periods up to and including 8 hours works fine. The jump to 1 day results in a 500.

By stepping back through logs using the absolute timeframe I was able to pin it down to 2017-05-23 19:00:06. Any attempt to use the “Quick values” function on the response_status field prior to that time on any of the nginx streams results in a 500.

This is super bizarre because this widget has been on these dashboards for three months at this point with no issues. Moreover, the actual response_status fields from the logs prior to that date appear just fine, it’s only the Quick values functionality that’s broken.

Any thoughts on this would be welcomed as I’m really at a loss to even come up with a mechanism by which this could happen.

[rtanay (Ryan Tanay)]

Update, as of today, now the death time is 2017-05-24 19:00:06

So it seems that there’s now some kind of rolling error where selecting any output prior to 19:00:06 of the preceding day causes Quick values on that single field to throw a 500.

[rtanay (Ryan Tanay)]

Further information:
Quick values work for any timeframe before 2017-05-23 19:00:06
Quick values work for any timeframe after 2017-05-24 19:00:06
Quick values do not work for any timeframe between those two points, or for any timeframe that overlaps any time between those two points.

Of note, the times & dates listed are in my local time, which is currently 5 hours offset to UTC, so it’s likely that something weird happened to that days index and is ruining all of that particular query that retrieves data from that index.

[jan (Jan Doberstein)]

Hej Ryan,

if you can name the time that exact, you should investigate the elasticsearch indexes. Look at the Field you like to get the quickvalue.

The error you are describing often happens if the field you like to work with contains different data types.

[rtanay (Ryan Tanay)]

Looks like that was the issue, thanks!

I knew that “generate chart” would blow up if fed data that contained non-numerics but I didn’t realize that disparate data types would blow up quick values as well.