Just upgraded to latest version and graylog stopped working

I upgraded to latest 3.2 yesterday, now when I click search it hangs on " Loading…", same when I click show received messages on inputs. Clicking on any dashboard makes the browser go crazy and flicker between pages unavailable, license missing and something. This happens in firefox and chrome, after clearning cache, rebooting the server and whatnot.

Hey @kthordarson,

can you please take a look in your graylog server.log for errors?

If you do not find any errors please have look in the web development console of firefox (this can be opened with F12).

This is from the server log:

There was an error fetching a resource: cannot GET https://172.16.4.99:9000/api/plugins/org.graylog.plugins.license/licenses/status/for-subject?subject=/license/enterprise/views (404). Additional information: No license status for subject /license/enterprise/views FetchProvider.js:19:14
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://172.16.4.99:9000/api/views/search/metadata. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).
There was an error fetching a resource: Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc… Additional information: Not available FetchProvider.js:19:14
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://172.16.4.99:9000/api/views/search/5e384d97aa67e925f2136245/execute. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).
There was an error fetching a resource: Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc… Additional information: Not available FetchProvider.js:19:14
Error: “Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc.”
r FetchProvider.js:9
t FetchProvider.js:13
request FetchProvider.js:87
s bluebird.js:5256
_settlePromiseFromHandler bluebird.js:3278
_settlePromise bluebird.js:3335
_settlePromise0 bluebird.js:3380
_settlePromises bluebird.js:3455
_drainQueue bluebird.js:185
_drainQueues bluebird.js:195
drainQueues bluebird.js:69
SearchStore.js:27:10

And this is from firefox:

There was an error fetching a resource: cannot GET https://172.16.4.99:9000/api/plugins/org.graylog.plugins.license/licenses/status/for-subject?subject=/license/enterprise/views (404). Additional information: No license status for subject /license/enterprise/views 2 FetchProvider.js:19:14
There was an error fetching a resource: cannot GET https://172.16.4.99:9000/api/plugins/org.graylog.plugins.license/licenses/status/for-subject?subject=/license/enterprise/views (404). Additional information: No license status for subject /license/enterprise/views FetchProvider.js:19:14
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://172.16.4.99:9000/api/views/search/metadata. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).
There was an error fetching a resource: Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc… Additional information: Not available FetchProvider.js:19:14
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://172.16.4.99:9000/api/views/search/5e384d97aa67e925f2136245/execute. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).
There was an error fetching a resource: Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc… Additional information: Not available FetchProvider.js:19:14
Error: “Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc.”
r FetchProvider.js:9
t FetchProvider.js:13
request FetchProvider.js:87
s bluebird.js:5256
_settlePromiseFromHandler bluebird.js:3278
_settlePromise bluebird.js:3335
_settlePromise0 bluebird.js:3380
_settlePromises bluebird.js:3455
_drainQueue bluebird.js:185
_drainQueues bluebird.js:195
drainQueues bluebird.js:69
SearchStore.js:27:10
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://172.16.4.99:9000/api/views/search/metadata. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).
There was an error fetching a resource: Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc… Additional information: Not available FetchProvider.js:19:14
Unhandled rejection r@https://172.16.4.99:9000/assets/plugin/org.graylog.plugins.enterprise.EnterprisePlugin/plugin.org.graylog.plugins.enterprise.EnterprisePlugin.bf017d1df900f969236c.js:164:110954
t@https://172.16.4.99:9000/assets/plugin/org.graylog.plugins.enterprise.EnterprisePlugin/plugin.org.graylog.plugins.enterprise.EnterprisePlugin.bf017d1df900f969236c.js:164:111829
value/this.request<@https://172.16.4.99:9000/assets/plugin/org.graylog.plugins.enterprise.EnterprisePlugin/plugin.org.graylog.plugins.enterprise.EnterprisePlugin.bf017d1df900f969236c.js:164:113585
s@https://172.16.4.99:9000/assets/plugin/org.graylog.plugins.enterprise.EnterprisePlugin/plugin.org.graylog.plugins.enterprise.EnterprisePlugin.bf017d1df900f969236c.js:207:305342
o/<[22]</n.exports/j.prototype._settlePromiseFromHandler@https://172.16.4.99:9000/assets/plugin/org.graylog.plugins.enterprise.EnterprisePlugin/plugin.org.graylog.plugins.enterprise.EnterprisePlugin.bf017d1df900f969236c.js:207:278624
o/<[22]</n.exports/j.prototype._settlePromise@https://172.16.4.99:9000/assets/plugin/org.graylog.plugins.enterprise.EnterprisePlugin/plugin.org.graylog.plugins.enterprise.EnterprisePlugin.bf017d1df900f969236c.js:207:279424
o/<[22]</n.exports/j.prototype._settlePromise0@https://172.16.4.99:9000/assets/plugin/org.graylog.plugins.enterprise.EnterprisePlugin/plugin.org.graylog.plugins.enterprise.EnterprisePlugin.bf017d1df900f969236c.js:207:280123
o/<[22]</n.exports/j.prototype._settlePromises@https://172.16.4.99:9000/assets/plugin/org.graylog.plugins.enterprise.EnterprisePlugin/plugin.org.graylog.plugins.enterprise.EnterprisePlugin.bf017d1df900f969236c.js:207:281339
o/<[2]</u.prototype._drainQueue@https://172.16.4.99:9000/assets/plugin/org.graylog.plugins.enterprise.EnterprisePlugin/plugin.org.graylog.plugins.enterprise.EnterprisePlugin.bf017d1df900f969236c.js:207:235316
o/<[2]</u.prototype._drainQueues@https://172.16.4.99:9000/assets/plugin/org.graylog.plugins.enterprise.EnterprisePlugin/plugin.org.graylog.plugins.enterprise.EnterprisePlugin.bf017d1df900f969236c.js:207:235377
u/this.drainQueues@https://172.16.4.99:9000/assets/plugin/org.graylog.plugins.enterprise.EnterprisePlugin/plugin.org.graylog.plugins.enterprise.EnterprisePlugin.bf017d1df900f969236c.js:207:233600
bluebird.js:1545:20
There was an error fetching a resource: cannot GET https://172.16.4.99:9000/api/plugins/org.graylog.plugins.license/licenses/status/for-subject?subject=/license/enterprise/views (404). Additional information: No license status for subject /license/enterprise/views 3 FetchProvider.js:19:14
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://172.16.4.99:9000/api/views/search/metadata. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).
There was an error fetching a resource: Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc… Additional information: Not available FetchProvider.js:19:14
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://172.16.4.99:9000/api/views/search/5e384d97aa67e925f2136245/execute. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).
There was an error fetching a resource: Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc… Additional information: Not available FetchProvider.js:19:14
Error: “Request has been terminated
Possible causes: the network is offline, Origin is not allowed by Access-Control-Allow-Origin, the page is being unloaded, etc.”
r FetchProvider.js:9
t FetchProvider.js:13
request FetchProvider.js:87
s bluebird.js:5256
_settlePromiseFromHandler bluebird.js:3278
_settlePromise bluebird.js:3335
_settlePromise0 bluebird.js:3380
_settlePromises bluebird.js:3455
_drainQueue bluebird.js:185
_drainQueues bluebird.js:195
drainQueues bluebird.js:69
SearchStore.js:27:10
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://172.16.4.99:9000/api/views/search/5e384d97aa67e925f2136245/execute. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).
There was an error fetching a resource: Request has been terminated

Sorry wrong paste from the server log.

This is the right output from the logs:

2020-02-04T10:25:51.325+01:00 ERROR [ServerRuntime$Responder] An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection closed
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_242]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_242]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Caused by: java.io.IOException: Connection closed
at org.glassfish.grizzly.asyncqueue.TaskQueue.onClose(TaskQueue.java:331) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.onClose(AbstractNIOAsyncQueueWriter.java:501) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.closeConnection(TCPNIOTransport.java:402) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.NIOConnection.doClose(NIOConnection.java:647) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.NIOConnection$6.run(NIOConnection.java:613) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.DefaultSelectorHandler.execute(DefaultSelectorHandler.java:238) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.NIOConnection.terminate0(NIOConnection.java:607) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOConnection.terminate0(TCPNIOConnection.java:291) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:136) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:106) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:344) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:108) ~[graylog.jar:?]
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:526) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) ~[graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593) ~[graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573) ~[graylog.jar:?]
… 1 more
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_242]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_242]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_242]
at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_242]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468) ~[?:1.8.0_242]
at org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:149) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOUtils.writeSimpleBuffer(TCPNIOUtils.java:133) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:126) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:106) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:344) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:108) ~[graylog.jar:?]
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:526) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) ~[graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:593) ~[graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:573) ~[graylog.jar:?]
… 1 more

Updated to latest 3.2.1. Seems to be working again.

2 Likes

Sorry that I missed to answer you. I am happy that we could be able to address this issue. Thanks a lot for your help!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.