2021-05-17T15:44:15.865Z WARN [ProxiedResource] Unable to call http://10.x.x.x:9000/api/system/jobs on node <66553xxx-f387-41d5-adf9-98637ecexxxx>: connect timed out
2021-05-17T15:44:18.202Z WARN [ProxiedResource] Unable to call http://10.x.x.x:9000/api/system on node <66553xxx-f387-41d5-adf9-98637ecexxxx>: connect timed out
2021-05-17T15:44:20.482Z WARN [ProxiedResource] Unable to call http://10.x.x.x:9000/api/system/metrics/multiple on node <66553xxx-f387-41d5-adf9-98637ecexxxx>: connect timed out
This could be a couple different issues like firewall is blocking connection, Selinux interfering with graylog, wrong configuration in Graylog server config file.
To help troubleshoot your issue we may need some more details about your environment.
If you’re unsure what you need then this may enlighten you.
@shoothub below is the command i am executing by getting into the server
command: echo -n “Hello Graylog, let’s be friends.” | nc -w 1 -u 127.0.0.1 9099
@shoothub I have mentioned the server logs in my question and i am writing it again, please find the below server logs
Server Logs:
2021-05-18T10:06:29.594Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T10:06:30.601Z WARN [ProxiedResource] Unable to call http://10.x.x.x:9000/api/system/metrics/multiple on node : connect timed out
2021-05-18T10:06:30.607Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T10:06:31.604Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T10:06:32.606Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T10:06:33.606Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T10:06:34.507Z WARN [ProxiedResource] Unable to call http://10.x.x.x:9000/api/system on node : connect timed out
2021-05-18T10:06:34.598Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T10:06:35.600Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
@shoothub I changed the binding the address and publish uri as I was confused with the NAT IP.
Below is the current error and am not able to stream the logs,
Serverlogs:
2021-05-18T10:45:32.254Z ERROR [ServerRuntime$Responder] An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Connection closed
at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:67) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139) ~[graylog.jar:?]
at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116) ~[graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:638) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:371) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:361) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?]
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) [graylog.jar:?]
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) [graylog.jar:?]
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?]
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
Caused by: java.io.IOException: Connection closed
at org.glassfish.grizzly.asyncqueue.TaskQueue.onClose(TaskQueue.java:307) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.onClose(AbstractNIOAsyncQueueWriter.java:477) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.closeConnection(TCPNIOTransport.java:379) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.NIOConnection.doClose(NIOConnection.java:643) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.NIOConnection$6.run(NIOConnection.java:609) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.DefaultSelectorHandler.execute(DefaultSelectorHandler.java:213) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.NIOConnection.terminate0(NIOConnection.java:603) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOConnection.terminate0(TCPNIOConnection.java:267) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:112) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:82) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:320) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:84) ~[graylog.jar:?]
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:515) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:79) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:66) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:391) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:360) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:324) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:255) ~[graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) ~[graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) ~[graylog.jar:?]
… 1 more
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_292]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_292]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_292]
at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_292]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470) ~[?:1.8.0_292]
at org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:125) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOUtils.writeCompositeBuffer(TCPNIOUtils.java:64) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:105) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:82) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:320) ~[graylog.jar:?]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:84) ~[graylog.jar:?]
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:515) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:79) ~[graylog.jar:?]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:66) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:391) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:360) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:324) ~[graylog.jar:?]
at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:255) ~[graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) ~[graylog.jar:?]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) ~[graylog.jar:?]
… 1 more
2021-05-18T10:45:32.338Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T10:45:32.726Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
http_bind_address = 10.x.x.x:9000 ( local ip address )
http_publish_uri = http://10.x.x.x:9000/ ( NAT ip address )
http_external_uri = http://10.x.x.x:9000/ ( local ip address )
http_enable_cors = false
After:
http_bind_address = 0.0.0.0:9000
http_publish_uri = http://10.x.x.x:9000/ ( NAT ip address )
http_external_uri = http://10.x.x.x:9000/ ( local ip address )
http_enable_cors = true
Currently the server.log has the below logs:
2021-05-18T11:18:51.648Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T11:18:52.644Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T11:18:53.652Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T11:18:54.645Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T11:18:55.640Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T11:18:56.641Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T11:18:57.644Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T11:18:58.645Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T11:18:59.645Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.
2021-05-18T11:19:55.832Z ERROR [AuditLogger] Unable to write audit log entry because there is no valid license.