Still experiencing Dashboard issue after 3.2 upgrade

Hello,

I wanted to bring this issue back up to see if there are more details on the solution.
https://community.graylog.org/t/solution-to-dashboard-issue-after-3-2-upgrade

I have tried using Robo3T(in text mode) and even had our DBAs to take a look, but we were not able to find any null time row config in the views collection. The DBAs were curious if there is a MongoDB query that can used to find the affected value? When trying to use the following query, we do not get any results.

where quantity is a positive number and unit [smhdwM]

We really would like to upgrade, but unfortunately this is a blocker as we can not simply drop our collection and start over.

Thanks!

@konrad Please let me know if you have any updates or can provide some help with this issue.

Hi @jamiebuxxx,

my morning today was trying to find a solution. The problem is, we would need a database pre 3.2 to see why this happens. We still do not know. But here is a query to find the culprit:

db.getCollection('searches').find({ "queries.search_types.row_groups.interval.timeunit": { $exists: true, $ne: "auto", $not: /^\d+[smhdwM]$/ } })

I thought the problem would be in the views table but was in searches. But with the query above you should be able to fix the broken search and the dashboards should load again. PLEASE let me know what you find inside that search so we can fix our migration perhaps.

Let me know if something is left unclear.

Best regards,
Konrad

Hi @konrad,

Thanks for the info. I just tried testing out the query on our nonpord cluster(running 3.2.6) and our prod cluster(running 3.1.4) and was unable find any results.

We are still seeing the following error when trying to access the dashboard on the dev Graylog cluster(3.2.6):

 Fetching dashboards failed with status: Error: cannot GET https://graylog-srv/api/views/dashboards?query=&page=1&per_page=10&sort=title&order=asc (500)

Please let me know if there is anything else I can try to get this working without losing the dashboard data. Thanks!

Cheers,
JaMile

Can you look if you see something in your graylog server log?

- Konrad

Hi @konrad,

Here’s what I found in the logs when I go to the dashboard page. I redacted some parts due to team and user info, but I the gist is there. It seems like a view or dashboard is missing??

2020-06-30T18:15:19.658Z ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.IllegalStateException: Search 5e711de246d556583b8f5a55 for view ViewDTO{id=5874180227cb3074457ac029, type=DASHBOARD, title=Team Name, summary=This dashboard was migrated automatically., description=Team Name error logs monitoring, searchId=5e711de246d556583b8f5a55, properties=[], requires={}, state={00000170-e9dc-bd97-84f4-123554884c99=ViewStateDTO{fields=null, staticMessageListId=null, titles=Titles{titles={widget={00000170-e9dc-bd91-84f4-123554884c99=TeamServiceName Unknown exceptions, 00000170-e9dc-bd90-84f4-123554884c99=TeamServiceName Access to the path is denied, 00000170-e9dc-bd8f-84f4-123554884c99=TeamServiceName Transaction was deadlocked, 00000170-e9dc-bd8e-84f4-123554884c99=Search result count}, tab={title=Team Name}}}, widgets=[WidgetDTO{id=00000170-e9dc-bd8e-84f4-123554884c99, type=aggregation, filter=null, timerange=Optional[RelativeRange{type=relative, range=28800}], query=Optional[elasticsearch: "ERROR: Unhandled exception" AND (NOT "deadlock") AND (NOT "Access to the path") AND app_name:TeamServiceName], streams=[], config=AggregationConfigDTO{rowPivots=[], columnPivots=[], series=[SeriesDTO{config=SeriesConfigDTO{name=null}, function=count()}], sort=[], visualization=numeric, visualizationConfig=NumberVisualizationConfigDTO{trend=true, trendPreference=HIGHER}, formattingSettings=null, rollup=true, eventAnnotation=false}}, WidgetDTO{id=00000170-e9dc-bd90-84f4-123554884c99, type=aggregation, filter=null, timerange=Optional[RelativeRange{type=relative, range=28800}], query=Optional[elasticsearch: "ERROR: Unhandled exception" AND "Access to the path" AND "is denied" AND app_name:TeamServiceName], streams=[], config=AggregationConfigDTO{rowPivots=[PivotDTO{field=timestamp, type=time, config=TimeHistogramConfigDTO{interval=TimeUnitIntervalDTO{type=timeunit, value=1, unit=minutes}}}], columnPivots=[], series=[SeriesDTO{config=SeriesConfigDTO{name=Messages}, function=count()}], sort=[], visualization=bar, visualizationConfig=null, formattingSettings=null, rollup=true, eventAnnotation=false}}, WidgetDTO{id=00000170-e9dc-bd8f-84f4-123554884c99, type=aggregation, filter=null, timerange=Optional[RelativeRange{type=relative, range=28800}], query=Optional[elasticsearch: "ERROR: Unhandled exception" AND "deadlock" AND app_name:TeamServiceName], streams=[], config=AggregationConfigDTO{rowPivots=[PivotDTO{field=timestamp, type=time, config=TimeHistogramConfigDTO{interval=TimeUnitIntervalDTO{type=timeunit, value=1, unit=minutes}}}], columnPivots=[], series=[SeriesDTO{config=SeriesConfigDTO{name=Messages}, function=count()}], sort=[], visualization=bar, visualizationConfig=null, formattingSettings=null, rollup=true, eventAnnotation=false}}, WidgetDTO{id=00000170-e9dc-bd91-84f4-123554884c99, type=aggregation, filter=null, timerange=Optional[RelativeRange{type=relative, range=28800}], query=Optional[elasticsearch: "ERROR: Unhandled exception" AND (NOT "deadlock") AND (NOT "Access to the path") AND app_name:TeamServiceName], streams=[], config=AggregationConfigDTO{rowPivots=[PivotDTO{field=timestamp, type=time, config=TimeHistogramConfigDTO{interval=TimeUnitIntervalDTO{type=timeunit, value=1, unit=minutes}}}], columnPivots=[], series=[SeriesDTO{config=SeriesConfigDTO{name=Messages}, function=count()}], sort=[], visualization=bar, visualizationConfig=null, formattingSettings=null, rollup=true, eventAnnotation=false}}], widgetMapping={00000170-e9dc-bd91-84f4-123554884c99=[00000170-e9dc-bd96-84f4-123554884c99], 00000170-e9dc-bd90-84f4-123554884c99=[00000170-e9dc-bd95-84f4-123554884c99], 00000170-e9dc-bd8f-84f4-123554884c99=[00000170-e9dc-bd94-84f4-123554884c99], 00000170-e9dc-bd8e-84f4-123554884c99=[00000170-e9dc-bd92-84f4-123554884c99, 00000170-e9dc-bd93-84f4-123554884c99]}, widgetPositions={00000170-e9dc-bd90-84f4-123554884c99=WidgetPositionDTO{col=org.graylog.plugins.views.search.views.NumberPosition@1f3f3ef6, row=org.graylog.plugins.views.search.views.NumberPosition@68fae399, height=org.graylog.plugins.views.search.views.NumberPosition@590beef6, width=org.graylog.plugins.views.search.views.NumberPosition@7c086d42}}, formatting=null, displayModeSettings=DisplayModeSettings{positions={}}}}, owner=Optional[Username], createdAt=2017-01-09T23:08:50.410Z} is missing.
	at org.graylog.plugins.views.search.views.RequiresParameterSupport.lambda$test$0(RequiresParameterSupport.java:45) ~[graylog.jar:?]
	at java.util.Optional.orElseThrow(Optional.java:290) ~[?:1.8.0_252]
	at org.graylog.plugins.views.search.views.RequiresParameterSupport.test(RequiresParameterSupport.java:45) ~[graylog.jar:?]
	at org.graylog.plugins.views.search.views.RequiresParameterSupport.test(RequiresParameterSupport.java:31) ~[graylog.jar:?]
	at org.graylog.plugins.views.Requirements.lambda$new$0(Requirements.java:39) ~[graylog.jar:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_252]
	at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_252]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_252]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_252]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_252]
	at org.graylog.plugins.views.Requirements.<init>(Requirements.java:41) ~[graylog.jar:?]
	at org.graylog.plugins.views.search.views.ViewRequirements.<init>(ViewRequirements.java:31) ~[graylog.jar:?]
	at org.graylog.plugins.views.search.views.ViewRequirements$$FastClassByGuice$$97cefd36.newInstance(<generated>) ~[graylog.jar:?]
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) ~[graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) ~[graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) ~[graylog.jar:?]
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) ~[graylog.jar:?]
	at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050) ~[graylog.jar:?]
	at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:836) ~[graylog.jar:?]
	at com.sun.proxy.$Proxy140.create(Unknown Source) ~[?:?]
	at org.graylog.plugins.views.search.views.ViewService.requirementsForView(ViewService.java:153) ~[graylog.jar:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_252]
	at java.util.ArrayList$Itr.forEachRemaining(ArrayList.java:899) ~[?:1.8.0_252]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_252]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_252]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_252]
	at org.graylog.plugins.views.search.views.ViewService.searchPaginated(ViewService.java:67) ~[graylog.jar:?]
	at org.graylog.plugins.views.search.views.ViewService.searchPaginatedByType(ViewService.java:86) ~[graylog.jar:?]
	at org.graylog.plugins.views.search.rest.DashboardsResource.views(DashboardsResource.java:89) ~[graylog.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]

If that is the case, is it possible to just find that record in the views collection and delete it? I would rather delete one view to save them all. :slight_smile:

Cheers,
JaMile

Hi @jamiebuxxx,

yes as it seems Team Name misses its search. There is not much you can do other than delete it.

You can do that by going to your mongo shell and execute:
> db.views.deleteOne({ "_id": ObjectId("5874180227cb3074457ac029") });

I hope this helps you.

Best regards,
Konrad

Hi @konrad,

Apologies for the delay, but I finally got back around to looking into this issue. After deleting the specific view with the mongo command, the Dashboard view is still giving an error. I’m also seeing the same error message in the logs, but it’s referencing a different ID.

I can keep trying to delete the IDs until the error message goes away. But even though I’m testing in dev, I don’t want to delete all the Dashboards and start over. Please let me know if you have any other ideas.

One thing we tried when having issues with the Alerts page(which was very similar to this), was exported the MongoDB collection for the alerts, deleted the alert collection from Mongo, modified the JSON to remove date and ID fields, and then used the Graylog API to add each alert back. That worked out fine!

However, when I tried to do this with the views, it was giving errors in the JSON structure when trying to remove fields that seemed specific to the entry(IDs, dates, etc). I would be willing to try export/import method again if you think that would work. Hopefully there’s some insight with JSON structure so I can make I’m removing the correct fields before importing.

Thanks for your help with this!!

Cheers,
JaMile

Hi @konrad,

Just wanted to update this thread so I won’t go stale and close. When you have a moment, could let me know if exporting and reimporting the dashboard JSON data back to Graylog is a doable idea. If so, what values should be removed the JSON before importing. Thanks!

Best Regards,
JaMile

Hi @jamiebuxxx,

I was a little engaged the past days in development. You have a problem I am unable to grasp without proper data. Is there a way that you could share you database before upgrade with me? I actually only need the dashboards?

Best regards,
Konrad

Hi @konrad,

I understand, it been a busy week for me as well. I don’t have an issue with sharing our database. I will message you privately to share the DB table with you.

Best,
JaMile