Next step after graylog configuration


(Long Tran) #1

Hello all,

Would you please guide me to the next step of being able to get log from windows workstations?

I am at where I am able to access the web interface of graylog.

In my environment, there are linux workstations, windows workstations, windows servers (AD), CentOS servers for nagios, and pfsense as firewall and routers.

I have read nxlog, but I might lack of fundamental knowledge for nxlog, that things do not make much sense for me.

I followed an instruction from this page https://www.supinfo.com/articles/single/6331-how-to-send-windows-server-2012-r2-logs-to-graylog-server
However, after I click on “Show received messages”, I got this error
“Error Message:
Unable to execute search
Exception:
org.elasticsearch.action.search.SearchPhaseExecutionException”

Please let me know if you would like more information, I am not sure what information you all would need.

Thank you very much.


(Jochen) #2

What’s in the logs of your Elasticsearch and Graylog nodes?
:arrow_right: http://docs.graylog.org/en/2.4/pages/configuration/file_location.html


(Long Tran) #3

Jochen,

The latest log is:

[2018-06-11 09:33:01, 692][INFO ] [cluster.service               ] [Wolf] remove{{graylog-06aaad24-b719-45e3-b2c0-4938d96ca6e0}{7_co-uM-T42oVLvoM92HTA}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false},}, reason:zen-disco-join{join from node[{graylog-06aaad24-b719-45e3-b2c0-4938d96ca6e0}{{7_co-uM-T42oVLvoM92HTA}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false}])
[2018-06-11 09:33:01, 692][INFO ] [cluster.service               ] [Wolf] added {{graylog-06aaad24-b719-45e3-b2c0-4938d96ca6e0}{7_co-uM-T42oVLvoM92HTA}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false},}, reason:zen-disco-join{join from node[{graylog-06aaad24-b719-45e3-b2c0-4938d96ca6e0}{{7_co-uM-T42oVLvoM92HTA}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false}])

The whole log file is filled with the above statements

Latest graylog log file says:
[SearchResource] Unable to execute search: all shards failed

Thank you for your help


(Jochen) #4

Maybe you could provide a little more context. :roll_eyes:


(Long Tran) #5

Sorry about that


(Jochen) #6

Please post the complete logs as text file (either inline or via a pastebin service such as https://0bin.net).


(Long Tran) #8
2018-06-12T10:22:55.230-05:00 WARN  [netty] [graylog-06aaad24-b719-45e3-b2c0-4938d96ca6e0] exception caught on transport layer [[id: 0x87002e55]], closing connection
java.net.SocketException: Network is unreachable
	at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_171]
	at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_171]
	at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_171]
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_171]
	at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108) [graylog.jar:?]
	at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70) [graylog.jar:?]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) [graylog.jar:?]
	at org.jboss.netty.channel.Channels.connect(Channels.java:634) [graylog.jar:?]
	at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:216) [graylog.jar:?]
	at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229) [graylog.jar:?]
	at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182) [graylog.jar:?]
	at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:949) [graylog.jar:?]
	at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916) [graylog.jar:?]
	at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888) [graylog.jar:?]
	at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267) [graylog.jar:?]
	at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395) [graylog.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
2018-06-12T10:22:56.730-05:00 WARN  [netty] [graylog-06aaad24-b719-45e3-b2c0-4938d96ca6e0] exception caught on transport layer [[id: 0x21fa8d66]], closing connection
java.net.SocketException: Network is unreachable
	at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_171]
	at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_171]
	at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_171]
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_171]
	at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108) [graylog.jar:?]
	at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70) [graylog.jar:?]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) [graylog.jar:?]
	at org.jboss.netty.channel.Channels.connect(Channels.java:634) [graylog.jar:?]
	at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:216) [graylog.jar:?]
	at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229) [graylog.jar:?]
	at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182) [graylog.jar:?]
	at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:949) [graylog.jar:?]
	at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916) [graylog.jar:?]
	at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888) [graylog.jar:?]
	at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267) [graylog.jar:?]
	at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395) [graylog.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
2018-06-12T10:22:56.731-05:00 WARN  [netty] [graylog-06aaad24-b719-45e3-b2c0-4938d96ca6e0] exception caught on transport layer [[id: 0xeb04889e]], closing connection
java.net.SocketException: Network is unreachable
	at sun.nio.ch.Net.connect0(Native Method) ~[?:1.8.0_171]
	at sun.nio.ch.Net.connect(Net.java:454) ~[?:1.8.0_171]
	at sun.nio.ch.Net.connect(Net.java:446) ~[?:1.8.0_171]
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[?:1.8.0_171]
	at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108) [graylog.jar:?]
	at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70) [graylog.jar:?]
	at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574) [graylog.jar:?]
	at org.jboss.netty.channel.Channels.connect(Channels.java:634) [graylog.jar:?]
	at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:216) [graylog.jar:?]
	at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:229) [graylog.jar:?]
	at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:182) [graylog.jar:?]
	at org.elasticsearch.transport.netty.NettyTransport.connectToChannelsLight(NettyTransport.java:949) [graylog.jar:?]
	at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyTransport.java:916) [graylog.jar:?]
	at org.elasticsearch.transport.netty.NettyTransport.connectToNodeLight(NettyTransport.java:888) [graylog.jar:?]
	at org.elasticsearch.transport.TransportService.connectToNodeLight(TransportService.java:267) [graylog.jar:?]
	at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing$3.run(UnicastZenPing.java:395) [graylog.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
2018-06-12T10:22:57.636-05:00 INFO  [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2018-06-12T10:22:59.437-05:00 INFO  [service] [graylog-06aaad24-b719-45e3-b2c0-4938d96ca6e0] detected_master {Freak of Science}{c-sN3OLgSAKMWE_7mSGHdg}{127.0.0.1}{127.0.0.1:9300}, added {{Freak of Science}{c-sN3OLgSAKMWE_7mSGHdg}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [{Freak of Science}{c-sN3OLgSAKMWE_7mSGHdg}{127.0.0.1}{127.0.0.1:9300}])
2018-06-12T10:23:23.159-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T11:20:52.873-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T11:51:06.607-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T12:21:22.655-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T12:51:35.721-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T13:21:51.074-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T13:52:06.203-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T14:22:21.591-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T14:52:35.747-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T15:22:50.897-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T15:53:06.217-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T16:23:21.423-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T16:53:35.445-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T17:23:50.574-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T17:54:04.488-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T18:24:19.613-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T18:54:34.704-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T19:24:49.878-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T19:55:03.910-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T20:25:19.157-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T20:55:34.198-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T21:25:51.309-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T21:56:05.493-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T22:26:18.159-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T22:56:33.510-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T23:26:50.546-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-12T23:57:05.942-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-13T00:27:18.192-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-13T00:57:36.859-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-13T01:27:47.954-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-13T01:58:04.221-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-13T02:28:17.219-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-13T02:58:31.487-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-13T03:59:02.960-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-13T04:29:16.977-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-13T04:59:33.192-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-13T05:29:48.392-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-13T06:00:01.202-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-13T06:30:17.462-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-13T07:00:31.054-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-13T07:30:44.785-05:00 WARN  [SearchResource] Unable to execute search: all shards failed
2018-06-13T08:01:00.175-05:00 WARN  [SearchResource] Unable to execute search: all shards failed

The forum and 0bin.net did not allow me to upload the whole file. Or I do not know how forum works to be able to share a 8MB log file. The log is yesterday and today


(Jochen) #9

The network connection between Graylog and Elasticsearch is unreliable or incorrectly configured.


(system) #10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.