Graylog startup failed - JerseyService FAILED

[DietmarSchurr]

Since the update to Graylog 5.0.6 the Graylog Service on one of two nodes is toggling, that means starting and stopping again with the following error message:

2023-04-14T14:37:07.045+02:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {FAILED=[JerseyService [FAILED]]}
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:769) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:581) ~[graylog.jar:?]
at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:295) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:273) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:324) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:260) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:45) [graylog.jar:?]
Suppressed: com.google.common.util.concurrent.ServiceManager$FailedService: JerseyService [FAILED]
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
at sun.nio.ch.Net.bind(Unknown Source) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.netBind(Unknown Source) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source) ~[?:?]
at sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source) ~[?:?]
at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bindToChannelAndAddress(TCPNIOBindingHandler.java:107) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bind(TCPNIOBindingHandler.java:64) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:215) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:195) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:186) ~[graylog.jar:?]
at org.glassfish.grizzly.http.server.NetworkListener.start(NetworkListener.java:711) ~[graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:256) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUpApi(JerseyService.java:205) ~[graylog.jar:?]
at org.graylog2.shared.initializers.JerseyService.startUp(JerseyService.java:159) ~[graylog.jar:?]
at com.google.common.util.concurrent.AbstractIdleService$DelegateService$1.run(AbstractIdleService.java:62) ~[graylog.jar:?]
at com.google.common.util.concurrent.Callables$4.run(Callables.java:121) ~[graylog.jar:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
2023-04-14T14:37:07.055+02:00 INFO [Server] SIGNAL received. Shutting down.

May be not the update to Graylog 5.0.6 (from 5.0.5) ist the problem, rather a background update of the SLES 15 to level SLES15 SP5 which was done by the “provider”.

There is a two node Opensearch cluster running in the background (2.5.0).

The strange thing is, that one node (master) runs fine, and the other one (not master) has the mentioned problem.

Does anyone know waht to do or check?

Regards,

Dietmar Schurr

[dennis]

Hey Dietmar!

The error is resulting from the Graylog server failing to bind to a port for the HTTP interface. The most common error would be that there is already something running on port 9000. Can you make sure that no other process is occupying that port?

[DietmarSchurr]

The port seems to be unoccupied.
netstat -tulpen | grep 9000
gives no result.

I checked meanwhile SELinux, seems to be disabled.

Thanks in advance for your help.

Dietmar Schurr

[DietmarSchurr]

Hello,

in fact there was the value for http_bind_address in the server.conf wrong.

Once I fixed it, the problem was gone.

Thanks for your help.

Regards,

Dietmar Schurr