Errors after upgrading from 2.5.2 to 3.1

[Werrex]

Greetings! I’d highly appreciate, if someone wiser here would be able to assist in a few issues I’m experiencing.

I updated our Graylog instance from 2.5.2 to the latest 3.1 following to the update documentation as closely as possible. Elasticsearch was already 5.6.13 thus no separate update for that required. In short, I’ve updated the server.conf with new settings as well as the nginx configuration.

Graylog loads up and I’m able to access it and various settings and the dashboards we have in use. However, I’m unable to open the System>Nodes>our node ID. This results in a HTTP 500 error code with the following error messages:

Could not get plugins: 
Getting plugins on node "<node ID>" failed: Error: cannot GET https://<FQDN>/api/cluster/<node ID>/plugins (500)
Could not get JVM information
Getting JVM information for node '<node ID>' failed: Error: cannot GET https://<FQDN>/api/cluster/<node ID>/jvm (500)

I’m also unable to start any inputs, resulting in the following error message and log entry:

Input '<input name>' could not be started. Request to start input '<input name>' failed. Check your Graylog logs for more information. 
WARN  [ProxiedResource] Unable to call https://api/system/inputstates/<input ID> on node <<node ID>>: api 

At this point I also noticed that the server.log is getting spammed with similar error messages:

 WARN  [ProxiedResource] Unable to call https://api/system/metrics/multiple on node <<node ID>>: api

Correct me if I’m wrong, but there’s definitely something fishy going on if the system is trying to call to an address without the server IP or FQDN name? The IP/FQDN related parameters that I have in the server.conf are the following:

http_bind_address = 127.0.0.1:9000
http_publish_uri = https://$http_bind_address/
http_external_uri = <FQDN>/ (same issues persist even with this removed)
elasticsearch_hosts = http://127.0.0.1:9200

Considering that everything was working before updating from 2.5.2, what am I missing? Luckily I do have a snapshot from before the update, but it would be nice to have the new features, you know?

Thanks a ton in advance!

[jan (Jan Doberstein)]

he @Werrex

if we would know your rest_* and web_* settings from before we might be able to help you.

[Werrex]

Hi Jan, thanks for the quick reply. I’ve pasted the parameters from the configuration below.

As a note, all parameters were set to use HTTP prior and post-update I’ve tested both HTTP/HTTPS and w/wo the endpoint/external URI with no affect on outcome. Also, during update I kept the existing configuration file and manually updated the changed parameters and added the new bin_dir and data_dir parameters.

I can paste the full server.conf if necessary, although no other parameters were touched upon.

bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server

#rest_listen_uri = http://127.0.0.1:9000/api 
http_bind_address = 127.0.0.1:9000

#rest_transport_uri = http://127.0.0.1:9000/api
http_publish_uri = https://$http_bind_address/

#web_listen_uri = http://127.0.0.1:9000/

#web_endpoint_uri =
http_external_uri = <hostname.domain>.net/

[jan (Jan Doberstein)]

if you use http_publish_uri = https://$http_bind_address/ did you enable https in the server.conf with a certificate and by setting?

In addition the http_external_uri = <hostname.domain>.net/ needs the protocol too …

---

I guess you have Graylog running on ONE server and you have a proxy like NGINX, Apache or similar that make it available to the external network.

[Werrex]

Yes, I’m running a single server with Graylog and Elasticsearch with nginx setup for external https access. The https and certificates are setup in the server.conf and I haven’t had any certificate issues prior or after the update, as I’m able to access Graylog portal via https with no issues.

I wasn’t aware of including the protocol to http_external_uri, that has now been fixed.

Unfortunately, perhaps expected, this did not help with the error messages I’m receiving. The below error messages appear every second or more frequently to the log.

 WARN  [ProxiedResource] Unable to call https://api/system/metrics/multiple on node <<node ID>>: api

I’ve come across other discussions on this forum with similar errors, but even those the errors contained an actual URI in the messages, not an https://api/system/...

While attempting to open the System > Nodes > nodeID, the following error is generated to the logs:

2019-09-09T18:16:57.267+03:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.net.UnknownHostException: api
at java.net.InetAddress.getAllByName0(InetAddress.java:1281) ~[?:1.8.0_222]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[?:1.8.0_222]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[?:1.8.0_222]
at okhttp3.Dns.lambda$static$0(Dns.java:39) ~[graylog.jar:?]
at okhttp3.internal.connection.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:171) ~[graylog.jar:?]
at okhttp3.internal.connection.RouteSelector.nextProxy(RouteSelector.java:135) ~[graylog.jar:?]
at okhttp3.internal.connection.RouteSelector.next(RouteSelector.java:84) ~[graylog.jar:?]
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:187) ~[graylog.jar:?]
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108) ~[graylog.jar:?]
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88) ~[graylog.jar:?]
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169) ~[graylog.jar:?]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:61) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[graylog.jar:?]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[graylog.jar:?]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:221) ~[graylog.jar:?]
at okhttp3.RealCall.execute(RealCall.java:81) ~[graylog.jar:?]
at retrofit2.OkHttpCall.execute(OkHttpCall.java:188) ~[graylog.jar:?]
at org.graylog2.rest.resources.cluster.ClusterSystemResource.jvm(ClusterSystemResource.java:90) ~[graylog.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]

[Werrex]

That is true, I’m running Graylog and Elasticsearch on the same server with nginx.

I also receive a Java related error when attempting to open the node settings. The post with this log message is pending review from our beloved moderators.

[jan (Jan Doberstein)]

just to clarify

http_publish_uri = https://$http_bind_address/

is written exactly like that? Just make a comment # before or use

so the default setting kicks in.

plus that this holds the nginx reachable address.

http_external_uri = https://<hostname.domain>/

disable ssl in graylog itself - or if you want/need that use https for the publish uri and take care that the certificates are trusted by Graylog.

[Werrex]

Thanks Jan, it seems that commenting out the following:

http_publish_uri = https://$http_bind_address/

Helped as all of the inputs now auto-started and the node is back online without any HTTP 500 errors. This with HTTPS and TLS certificates in place as before and the http_external_uri pointing to the nginx address as since yesterday.

To be honest, I’m not quite certain why commenting out that one parameter, which is part of the 3.x configuration documentation, would be the cause for this? Well, for now I’ll be testing the hell out of our Graylog in case there are any other issues or errors appearing.

Big thanks to you Jan!