Unable to view messages on new install

[gerryrigney]

Hi all

Have setup a Graylog server and for the most part it seems to be working however when I try to view logs they do not display (constant spinning “Loading” message). When I look in the system – overview there are no errors, everything (Indexer, Elasticsearch, Notifications) is showing up green. When I go to the messages either by going to my Inputs and selecting Show Messages for one of them, the following show up in the Graplog server log

2017-06-22T10:20:12.032+01:00 WARN [SearchResource] Unable to execute search: all shards failed

When I check the elasticsearch/graylog log at the same time there are no errors in there, the last log being when I restarted the services yesterday

[2017-06-21 15:06:59,685][INFO ][node                     ] [Triathlon] starting ...
[2017-06-21 15:06:59,817][INFO ][transport                ] [Triathlon] publish_address {172.20.10.12:9300}, bound_addresses {172.20.10.12:9300}
[2017-06-21 15:06:59,825][INFO ][discovery                ] [Triathlon] graylog/yeyPLA55T7G24H3GhK9u-A
[2017-06-21 15:07:02,865][INFO ][cluster.service          ] [Triathlon] new_master {Triathlon}{yeyPLA55T7G24H3GhK9u-A}{172.20.10.12}{172.20.10.12:9300}, added {{graylog-4e500f0d-eb65-45b1-9225-9452b329d6e4}{Ss44szpESsu1yYZQYrLSIw}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-06-21 15:07:02,914][INFO ][http                     ] [Triathlon] publish_address {172.20.10.12:9200}, bound_addresses {172.20.10.12:9200}
[2017-06-21 15:07:02,915][INFO ][node                     ] [Triathlon] started
[2017-06-21 15:07:02,997][INFO ][gateway                  ] [Triathlon] recovered [1] indices into cluster_state
[2017-06-21 15:07:03,905][INFO ][cluster.routing.allocation] [Triathlon] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
[2017-06-21 15:08:37,752][INFO ][cluster.service          ] [Triathlon] removed {{graylog-4e500f0d-eb65-45b1-9225-9452b329d6e4}{Ss44szpESsu1yYZQYrLSIw}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false},}, reason: zen-disco-node-left({graylog-4e500f0d-eb65-45b1-9225-9452b329d6e4}{Ss44szpESsu1yYZQYrLSIw}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false}), reason(left)
[2017-06-21 15:09:00,965][INFO ][cluster.service          ] [Triathlon] added {{graylog-4e500f0d-eb65-45b1-9225-9452b329d6e4}{JoimL-WjSICx0DZOJ68vuw}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-4e500f0d-eb65-45b1-9225-9452b329d6e4}{JoimL-WjSICx0DZOJ68vuw}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false}])

Any thoughts on where to check next? I also get the same error in the server if I go to streams to view or search for messages, doesn’t matter what timescale I put on it. When I look at the input node it shows 1 connection and indicates messages have been received, as does the server I have sending them (Windows server via nxlog)

Install details are:
Running CentOS 7 on a VM
6GB RAM (Heap set to 3GB)
Graylog 2.2.3
Elasticsearch 2.4.5
Mongo DB 3.4.5

Many thanks for any help or pointers!
gR

[jan (Jan Doberstein)]

did you check your journal? Are any messages in Elasticsearch? Is Graylog able to send messages to Elasticsearch?

[gerryrigney]

Hi Jan, thanks for the reply. No errors in journalctl that I can see. Just setting this up now so if I’ve done this wrong please let me know but when I check curl -XGET http://localhost:9200/_stats/docs,store Elasticsearch is showing 1000s of messages so I’m assuming its Graylog putting them in there as nothing else would be configured to do so

{"_shards":{"total":1,"successful":1,"failed":0},"_all":{"primaries":{"docs":{"count":28720,"deleted":0},"store":{"size_in_bytes":15615501,"throttle_time_in_millis":0}},"total":{"docs":{"count":28720,"deleted":0},"store":{"size_in_bytes":15615501,"throttle_time_in_millis":0}}},"indices":{"graylog_0":{"primaries":{"docs":{"count":28720,"deleted":0},"store":{"size_in_bytes":15615501,"throttle_time_in_millis":0}},"total":{"docs":{"count":28720,"deleted":0},"store":{"size_in_bytes":15615501,"throttle_time_in_millis":0}}}}}

Thanks for the tips, appreciate any more pointers you could provide,
gR

[jan (Jan Doberstein)]

@gerryrigney you should check your graylog server.log and your elasticsearch log. Where to find depends on the setup but defaults are documented here.

Did System > Overview in the Graylog Webinterface gives you an idea what the problem is?

[gerryrigney]

No obvious errors in the logs with the exception of the:
[SearchResource] Unable to execute search: all shards failed
in the graylog.log everytime I do a search or go to view messages. Everything is green in the overview (see attached)

Thanks
gR

[gerryrigney]

Just to note, clean install using the guide: http://docs.graylog.org/en/2.3/pages/installation/os/centos.html and messages are now showing up.
Thanks

[Panther2289 (santhosh)]

Hello,

I too face the same issue.

Search area displays “loading” - unable to fetch any logs.

But I’m sure that receiving logs from sources - please find the below reference screenshot

Note: All Time stamp are same.

regards,