Still experiencing Dashboard issue after 3.2 upgrade

Hello,

I wanted to bring this issue back up to see if there are more details on the solution.
https://community.graylog.org/t/solution-to-dashboard-issue-after-3-2-upgrade

I have tried using Robo3T(in text mode) and even had our DBAs to take a look, but we were not able to find any null time row config in the views collection. The DBAs were curious if there is a MongoDB query that can used to find the affected value? When trying to use the following query, we do not get any results.

where quantity is a positive number and unit [smhdwM]

We really would like to upgrade, but unfortunately this is a blocker as we can not simply drop our collection and start over.

Thanks!

@konrad Please let me know if you have any updates or can provide some help with this issue.

Hi @jamiebuxxx,

my morning today was trying to find a solution. The problem is, we would need a database pre 3.2 to see why this happens. We still do not know. But here is a query to find the culprit:

db.getCollection('searches').find({ "queries.search_types.row_groups.interval.timeunit": { $exists: true, $ne: "auto", $not: /^\d+[smhdwM]$/ } })

I thought the problem would be in the views table but was in searches. But with the query above you should be able to fix the broken search and the dashboards should load again. PLEASE let me know what you find inside that search so we can fix our migration perhaps.

Let me know if something is left unclear.

Best regards,
Konrad

Hi @konrad,

Thanks for the info. I just tried testing out the query on our nonpord cluster(running 3.2.6) and our prod cluster(running 3.1.4) and was unable find any results.

We are still seeing the following error when trying to access the dashboard on the dev Graylog cluster(3.2.6):

 Fetching dashboards failed with status: Error: cannot GET https://graylog-srv/api/views/dashboards?query=&page=1&per_page=10&sort=title&order=asc (500)

Please let me know if there is anything else I can try to get this working without losing the dashboard data. Thanks!

Cheers,
JaMile

Can you look if you see something in your graylog server log?

- Konrad

Hi @konrad,

Here’s what I found in the logs when I go to the dashboard page. I redacted some parts due to team and user info, but I the gist is there. It seems like a view or dashboard is missing??

2020-06-30T18:15:19.658Z ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.IllegalStateException: Search 5e711de246d556583b8f5a55 for view ViewDTO{id=5874180227cb3074457ac029, type=DASHBOARD, title=Team Name, summary=This dashboard was migrated automatically., description=Team Name error logs monitoring, searchId=5e711de246d556583b8f5a55, properties=[], requires={}, state={00000170-e9dc-bd97-84f4-123554884c99=ViewStateDTO{fields=null, staticMessageListId=null, titles=Titles{titles={widget={00000170-e9dc-bd91-84f4-123554884c99=TeamServiceName Unknown exceptions, 00000170-e9dc-bd90-84f4-123554884c99=TeamServiceName Access to the path is denied, 00000170-e9dc-bd8f-84f4-123554884c99=TeamServiceName Transaction was deadlocked, 00000170-e9dc-bd8e-84f4-123554884c99=Search result count}, tab={title=Team Name}}}, widgets=[WidgetDTO{id=00000170-e9dc-bd8e-84f4-123554884c99, type=aggregation, filter=null, timerange=Optional[RelativeRange{type=relative, range=28800}], query=Optional[elasticsearch: "ERROR: Unhandled exception" AND (NOT "deadlock") AND (NOT "Access to the path") AND app_name:TeamServiceName], streams=[], config=AggregationConfigDTO{rowPivots=[], columnPivots=[], series=[SeriesDTO{config=SeriesConfigDTO{name=null}, function=count()}], sort=[], visualization=numeric, visualizationConfig=NumberVisualizationConfigDTO{trend=true, trendPreference=HIGHER}, formattingSettings=null, rollup=true, eventAnnotation=false}}, WidgetDTO{id=00000170-e9dc-bd90-84f4-123554884c99, type=aggregation, filter=null, timerange=Optional[RelativeRange{type=relative, range=28800}], query=Optional[elasticsearch: "ERROR: Unhandled exception" AND "Access to the path" AND "is denied" AND app_name:TeamServiceName], streams=[], config=AggregationConfigDTO{rowPivots=[PivotDTO{field=timestamp, type=time, config=TimeHistogramConfigDTO{interval=TimeUnitIntervalDTO{type=timeunit, value=1, unit=minutes}}}], columnPivots=[], series=[SeriesDTO{config=SeriesConfigDTO{name=Messages}, function=count()}], sort=[], visualization=bar, visualizationConfig=null, formattingSettings=null, rollup=true, eventAnnotation=false}}, WidgetDTO{id=00000170-e9dc-bd8f-84f4-123554884c99, type=aggregation, filter=null, timerange=Optional[RelativeRange{type=relative, range=28800}], query=Optional[elasticsearch: "ERROR: Unhandled exception" AND "deadlock" AND app_name:TeamServiceName], streams=[], config=AggregationConfigDTO{rowPivots=[PivotDTO{field=timestamp, type=time, config=TimeHistogramConfigDTO{interval=TimeUnitIntervalDTO{type=timeunit, value=1, unit=minutes}}}], columnPivots=[], series=[SeriesDTO{config=SeriesConfigDTO{name=Messages}, function=count()}], sort=[], visualization=bar, visualizationConfig=null, formattingSettings=null, rollup=true, eventAnnotation=false}}, WidgetDTO{id=00000170-e9dc-bd91-84f4-123554884c99, type=aggregation, filter=null, timerange=Optional[RelativeRange{type=relative, range=28800}], query=Optional[elasticsearch: "ERROR: Unhandled exception" AND (NOT "deadlock") AND (NOT "Access to the path") AND app_name:TeamServiceName], streams=[], config=AggregationConfigDTO{rowPivots=[PivotDTO{field=timestamp, type=time, config=TimeHistogramConfigDTO{interval=TimeUnitIntervalDTO{type=timeunit, value=1, unit=minutes}}}], columnPivots=[], series=[SeriesDTO{config=SeriesConfigDTO{name=Messages}, function=count()}], sort=[], visualization=bar, visualizationConfig=null, formattingSettings=null, rollup=true, eventAnnotation=false}}], widgetMapping={00000170-e9dc-bd91-84f4-123554884c99=[00000170-e9dc-bd96-84f4-123554884c99], 00000170-e9dc-bd90-84f4-123554884c99=[00000170-e9dc-bd95-84f4-123554884c99], 00000170-e9dc-bd8f-84f4-123554884c99=[00000170-e9dc-bd94-84f4-123554884c99], 00000170-e9dc-bd8e-84f4-123554884c99=[00000170-e9dc-bd92-84f4-123554884c99, 00000170-e9dc-bd93-84f4-123554884c99]}, widgetPositions={00000170-e9dc-bd90-84f4-123554884c99=WidgetPositionDTO{col=org.graylog.plugins.views.search.views.NumberPosition@1f3f3ef6, row=org.graylog.plugins.views.search.views.NumberPosition@68fae399, height=org.graylog.plugins.views.search.views.NumberPosition@590beef6, width=org.graylog.plugins.views.search.views.NumberPosition@7c086d42}}, formatting=null, displayModeSettings=DisplayModeSettings{positions={}}}}, owner=Optional[Username], createdAt=2017-01-09T23:08:50.410Z} is missing.
	at org.graylog.plugins.views.search.views.RequiresParameterSupport.lambda$test$0(RequiresParameterSupport.java:45) ~[graylog.jar:?]
	at java.util.Optional.orElseThrow(Optional.java:290) ~[?:1.8.0_252]
	at org.graylog.plugins.views.search.views.RequiresParameterSupport.test(RequiresParameterSupport.java:45) ~[graylog.jar:?]
	at org.graylog.plugins.views.search.views.RequiresParameterSupport.test(RequiresParameterSupport.java:31) ~[graylog.jar:?]
	at org.graylog.plugins.views.Requirements.lambda$new$0(Requirements.java:39) ~[graylog.jar:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_252]
	at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_252]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_252]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_252]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_252]
	at org.graylog.plugins.views.Requirements.<init>(Requirements.java:41) ~[graylog.jar:?]
	at org.graylog.plugins.views.search.views.ViewRequirements.<init>(ViewRequirements.java:31) ~[graylog.jar:?]
	at org.graylog.plugins.views.search.views.ViewRequirements$$FastClassByGuice$$97cefd36.newInstance(<generated>) ~[graylog.jar:?]
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) ~[graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) ~[graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) ~[graylog.jar:?]
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) ~[graylog.jar:?]
	at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050) ~[graylog.jar:?]
	at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:836) ~[graylog.jar:?]
	at com.sun.proxy.$Proxy140.create(Unknown Source) ~[?:?]
	at org.graylog.plugins.views.search.views.ViewService.requirementsForView(ViewService.java:153) ~[graylog.jar:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_252]
	at java.util.ArrayList$Itr.forEachRemaining(ArrayList.java:899) ~[?:1.8.0_252]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_252]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_252]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_252]
	at org.graylog.plugins.views.search.views.ViewService.searchPaginated(ViewService.java:67) ~[graylog.jar:?]
	at org.graylog.plugins.views.search.views.ViewService.searchPaginatedByType(ViewService.java:86) ~[graylog.jar:?]
	at org.graylog.plugins.views.search.rest.DashboardsResource.views(DashboardsResource.java:89) ~[graylog.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]

If that is the case, is it possible to just find that record in the views collection and delete it? I would rather delete one view to save them all. :slight_smile:

Cheers,
JaMile

Hi @jamiebuxxx,

yes as it seems Team Name misses its search. There is not much you can do other than delete it.

You can do that by going to your mongo shell and execute:
> db.views.deleteOne({ "_id": ObjectId("5874180227cb3074457ac029") });

I hope this helps you.

Best regards,
Konrad