Still experiencing Dashboard issue after 3.2 upgrade

jamiebuxxx · June 24, 2020, 9:47pm

Hello,

I wanted to bring this issue back up to see if there are more details on the solution.
https://community.graylog.org/t/solution-to-dashboard-issue-after-3-2-upgrade

I have tried using Robo3T(in text mode) and even had our DBAs to take a look, but we were not able to find any null time row config in the views collection. The DBAs were curious if there is a MongoDB query that can used to find the affected value? When trying to use the following query, we do not get any results.

where quantity is a positive number and unit [smhdwM]

We really would like to upgrade, but unfortunately this is a blocker as we can not simply drop our collection and start over.

Thanks!

jamiebuxxx · June 24, 2020, 9:47pm

@konrad Please let me know if you have any updates or can provide some help with this issue.

konrad · June 25, 2020, 10:15am

Hi @jamiebuxxx,

my morning today was trying to find a solution. The problem is, we would need a database pre 3.2 to see why this happens. We still do not know. But here is a query to find the culprit:

db.getCollection('searches').find({ "queries.search_types.row_groups.interval.timeunit": { $exists: true, $ne: "auto", $not: /^\d+[smhdwM]$/ } })

I thought the problem would be in the views table but was in searches. But with the query above you should be able to fix the broken search and the dashboards should load again. PLEASE let me know what you find inside that search so we can fix our migration perhaps.

Let me know if something is left unclear.

Best regards,
Konrad

jamiebuxxx · June 25, 2020, 3:49pm

Hi @konrad,

Thanks for the info. I just tried testing out the query on our nonpord cluster(running 3.2.6) and our prod cluster(running 3.1.4) and was unable find any results.

We are still seeing the following error when trying to access the dashboard on the dev Graylog cluster(3.2.6):

 Fetching dashboards failed with status: Error: cannot GET https://graylog-srv/api/views/dashboards?query=&page=1&per_page=10&sort=title&order=asc (500)

Please let me know if there is anything else I can try to get this working without losing the dashboard data. Thanks!

Cheers,
JaMile

konrad · June 26, 2020, 7:27am

Can you look if you see something in your graylog server log?

- Konrad

jamiebuxxx · June 30, 2020, 6:34pm

Hi @konrad,

Here’s what I found in the logs when I go to the dashboard page. I redacted some parts due to team and user info, but I the gist is there. It seems like a view or dashboard is missing??

2020-06-30T18:15:19.658Z ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.IllegalStateException: Search 5e711de246d556583b8f5a55 for view ViewDTO{id=5874180227cb3074457ac029, type=DASHBOARD, title=Team Name, summary=This dashboard was migrated automatically., description=Team Name error logs monitoring, searchId=5e711de246d556583b8f5a55, properties=[], requires={}, state={00000170-e9dc-bd97-84f4-123554884c99=ViewStateDTO{fields=null, staticMessageListId=null, titles=Titles{titles={widget={00000170-e9dc-bd91-84f4-123554884c99=TeamServiceName Unknown exceptions, 00000170-e9dc-bd90-84f4-123554884c99=TeamServiceName Access to the path is denied, 00000170-e9dc-bd8f-84f4-123554884c99=TeamServiceName Transaction was deadlocked, 00000170-e9dc-bd8e-84f4-123554884c99=Search result count}, tab={title=Team Name}}}, widgets=[WidgetDTO{id=00000170-e9dc-bd8e-84f4-123554884c99, type=aggregation, filter=null, timerange=Optional[RelativeRange{type=relative, range=28800}], query=Optional[elasticsearch: "ERROR: Unhandled exception" AND (NOT "deadlock") AND (NOT "Access to the path") AND app_name:TeamServiceName], streams=[], config=AggregationConfigDTO{rowPivots=[], columnPivots=[], series=[SeriesDTO{config=SeriesConfigDTO{name=null}, function=count()}], sort=[], visualization=numeric, visualizationConfig=NumberVisualizationConfigDTO{trend=true, trendPreference=HIGHER}, formattingSettings=null, rollup=true, eventAnnotation=false}}, WidgetDTO{id=00000170-e9dc-bd90-84f4-123554884c99, type=aggregation, filter=null, timerange=Optional[RelativeRange{type=relative, range=28800}], query=Optional[elasticsearch: "ERROR: Unhandled exception" AND "Access to the path" AND "is denied" AND app_name:TeamServiceName], streams=[], config=AggregationConfigDTO{rowPivots=[PivotDTO{field=timestamp, type=time, config=TimeHistogramConfigDTO{interval=TimeUnitIntervalDTO{type=timeunit, value=1, unit=minutes}}}], columnPivots=[], series=[SeriesDTO{config=SeriesConfigDTO{name=Messages}, function=count()}], sort=[], visualization=bar, visualizationConfig=null, formattingSettings=null, rollup=true, eventAnnotation=false}}, WidgetDTO{id=00000170-e9dc-bd8f-84f4-123554884c99, type=aggregation, filter=null, timerange=Optional[RelativeRange{type=relative, range=28800}], query=Optional[elasticsearch: "ERROR: Unhandled exception" AND "deadlock" AND app_name:TeamServiceName], streams=[], config=AggregationConfigDTO{rowPivots=[PivotDTO{field=timestamp, type=time, config=TimeHistogramConfigDTO{interval=TimeUnitIntervalDTO{type=timeunit, value=1, unit=minutes}}}], columnPivots=[], series=[SeriesDTO{config=SeriesConfigDTO{name=Messages}, function=count()}], sort=[], visualization=bar, visualizationConfig=null, formattingSettings=null, rollup=true, eventAnnotation=false}}, WidgetDTO{id=00000170-e9dc-bd91-84f4-123554884c99, type=aggregation, filter=null, timerange=Optional[RelativeRange{type=relative, range=28800}], query=Optional[elasticsearch: "ERROR: Unhandled exception" AND (NOT "deadlock") AND (NOT "Access to the path") AND app_name:TeamServiceName], streams=[], config=AggregationConfigDTO{rowPivots=[PivotDTO{field=timestamp, type=time, config=TimeHistogramConfigDTO{interval=TimeUnitIntervalDTO{type=timeunit, value=1, unit=minutes}}}], columnPivots=[], series=[SeriesDTO{config=SeriesConfigDTO{name=Messages}, function=count()}], sort=[], visualization=bar, visualizationConfig=null, formattingSettings=null, rollup=true, eventAnnotation=false}}], widgetMapping={00000170-e9dc-bd91-84f4-123554884c99=[00000170-e9dc-bd96-84f4-123554884c99], 00000170-e9dc-bd90-84f4-123554884c99=[00000170-e9dc-bd95-84f4-123554884c99], 00000170-e9dc-bd8f-84f4-123554884c99=[00000170-e9dc-bd94-84f4-123554884c99], 00000170-e9dc-bd8e-84f4-123554884c99=[00000170-e9dc-bd92-84f4-123554884c99, 00000170-e9dc-bd93-84f4-123554884c99]}, widgetPositions={00000170-e9dc-bd90-84f4-123554884c99=WidgetPositionDTO{col=org.graylog.plugins.views.search.views.NumberPosition@1f3f3ef6, row=org.graylog.plugins.views.search.views.NumberPosition@68fae399, height=org.graylog.plugins.views.search.views.NumberPosition@590beef6, width=org.graylog.plugins.views.search.views.NumberPosition@7c086d42}}, formatting=null, displayModeSettings=DisplayModeSettings{positions={}}}}, owner=Optional[Username], createdAt=2017-01-09T23:08:50.410Z} is missing.
	at org.graylog.plugins.views.search.views.RequiresParameterSupport.lambda$test$0(RequiresParameterSupport.java:45) ~[graylog.jar:?]
	at java.util.Optional.orElseThrow(Optional.java:290) ~[?:1.8.0_252]
	at org.graylog.plugins.views.search.views.RequiresParameterSupport.test(RequiresParameterSupport.java:45) ~[graylog.jar:?]
	at org.graylog.plugins.views.search.views.RequiresParameterSupport.test(RequiresParameterSupport.java:31) ~[graylog.jar:?]
	at org.graylog.plugins.views.Requirements.lambda$new$0(Requirements.java:39) ~[graylog.jar:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_252]
	at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_252]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_252]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_252]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_252]
	at org.graylog.plugins.views.Requirements.<init>(Requirements.java:41) ~[graylog.jar:?]
	at org.graylog.plugins.views.search.views.ViewRequirements.<init>(ViewRequirements.java:31) ~[graylog.jar:?]
	at org.graylog.plugins.views.search.views.ViewRequirements$$FastClassByGuice$$97cefd36.newInstance(<generated>) ~[graylog.jar:?]
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) ~[graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) ~[graylog.jar:?]
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) ~[graylog.jar:?]
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) ~[graylog.jar:?]
	at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050) ~[graylog.jar:?]
	at com.google.inject.assistedinject.FactoryProvider2.invoke(FactoryProvider2.java:836) ~[graylog.jar:?]
	at com.sun.proxy.$Proxy140.create(Unknown Source) ~[?:?]
	at org.graylog.plugins.views.search.views.ViewService.requirementsForView(ViewService.java:153) ~[graylog.jar:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_252]
	at java.util.ArrayList$Itr.forEachRemaining(ArrayList.java:899) ~[?:1.8.0_252]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_252]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_252]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_252]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_252]
	at org.graylog.plugins.views.search.views.ViewService.searchPaginated(ViewService.java:67) ~[graylog.jar:?]
	at org.graylog.plugins.views.search.views.ViewService.searchPaginatedByType(ViewService.java:86) ~[graylog.jar:?]
	at org.graylog.plugins.views.search.rest.DashboardsResource.views(DashboardsResource.java:89) ~[graylog.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]

If that is the case, is it possible to just find that record in the views collection and delete it? I would rather delete one view to save them all.

Cheers,
JaMile

konrad · July 3, 2020, 9:02am

Hi @jamiebuxxx,

yes as it seems Team Name misses its search. There is not much you can do other than delete it.

You can do that by going to your mongo shell and execute:
> db.views.deleteOne({ "_id": ObjectId("5874180227cb3074457ac029") });

I hope this helps you.

Best regards,
Konrad

jamiebuxxx · July 10, 2020, 8:21pm

Hi @konrad,

Apologies for the delay, but I finally got back around to looking into this issue. After deleting the specific view with the mongo command, the Dashboard view is still giving an error. I’m also seeing the same error message in the logs, but it’s referencing a different ID.

I can keep trying to delete the IDs until the error message goes away. But even though I’m testing in dev, I don’t want to delete all the Dashboards and start over. Please let me know if you have any other ideas.

One thing we tried when having issues with the Alerts page(which was very similar to this), was exported the MongoDB collection for the alerts, deleted the alert collection from Mongo, modified the JSON to remove date and ID fields, and then used the Graylog API to add each alert back. That worked out fine!

However, when I tried to do this with the views, it was giving errors in the JSON structure when trying to remove fields that seemed specific to the entry(IDs, dates, etc). I would be willing to try export/import method again if you think that would work. Hopefully there’s some insight with JSON structure so I can make I’m removing the correct fields before importing.

Thanks for your help with this!!

Cheers,
JaMile

jamiebuxxx · July 17, 2020, 5:29pm

Hi @konrad,

Just wanted to update this thread so I won’t go stale and close. When you have a moment, could let me know if exporting and reimporting the dashboard JSON data back to Graylog is a doable idea. If so, what values should be removed the JSON before importing. Thanks!

Best Regards,
JaMile

konrad · July 24, 2020, 7:46am

Hi @jamiebuxxx,

I was a little engaged the past days in development. You have a problem I am unable to grasp without proper data. Is there a way that you could share you database before upgrade with me? I actually only need the dashboards?

Best regards,
Konrad

jamiebuxxx · July 31, 2020, 8:51pm

Hi @konrad,

I understand, it been a busy week for me as well. I don’t have an issue with sharing our database. I will message you privately to share the DB table with you.

Best,
JaMile

jamiebuxxx · August 14, 2020, 6:45pm

Hi @konrad,

Just wanted to keep this thread open. Was curious if you have any updates? Thanks again for your help with this.

Best regards,
JaMile

system · August 28, 2020, 6:46pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Solution to Dashboard issue after 3.2 upgrade Graylog Central (peer support)	3	1069	April 15, 2020
Dashboards migration from 3.1 to 3.2 Graylog Central (peer support) sidecar	5	827	May 26, 2021
All dashboards and Search Queries lost after to 3.2.1 Graylog Central (peer support)	4	839	March 2, 2020
Lost All dashboards after upgrade from 3.1.x to 3.3.5 Graylog Central (peer support)	5	667	October 1, 2020
All dashboards lost after upgrade from 3.1.x to 3.2.1 Graylog Central (peer support)	10	1155	March 2, 2020

Still experiencing Dashboard issue after 3.2 upgrade

Related Topics