Unable to view messages on new install


#1

Hi all

Have setup a Graylog server and for the most part it seems to be working however when I try to view logs they do not display (constant spinning “Loading” message). When I look in the system – overview there are no errors, everything (Indexer, Elasticsearch, Notifications) is showing up green. When I go to the messages either by going to my Inputs and selecting Show Messages for one of them, the following show up in the Graplog server log

2017-06-22T10:20:12.032+01:00 WARN [SearchResource] Unable to execute search: all shards failed

When I check the elasticsearch/graylog log at the same time there are no errors in there, the last log being when I restarted the services yesterday

[2017-06-21 15:06:59,685][INFO ][node                     ] [Triathlon] starting ...
[2017-06-21 15:06:59,817][INFO ][transport                ] [Triathlon] publish_address {172.20.10.12:9300}, bound_addresses {172.20.10.12:9300}
[2017-06-21 15:06:59,825][INFO ][discovery                ] [Triathlon] graylog/yeyPLA55T7G24H3GhK9u-A
[2017-06-21 15:07:02,865][INFO ][cluster.service          ] [Triathlon] new_master {Triathlon}{yeyPLA55T7G24H3GhK9u-A}{172.20.10.12}{172.20.10.12:9300}, added {{graylog-4e500f0d-eb65-45b1-9225-9452b329d6e4}{Ss44szpESsu1yYZQYrLSIw}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2017-06-21 15:07:02,914][INFO ][http                     ] [Triathlon] publish_address {172.20.10.12:9200}, bound_addresses {172.20.10.12:9200}
[2017-06-21 15:07:02,915][INFO ][node                     ] [Triathlon] started
[2017-06-21 15:07:02,997][INFO ][gateway                  ] [Triathlon] recovered [1] indices into cluster_state
[2017-06-21 15:07:03,905][INFO ][cluster.routing.allocation] [Triathlon] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_0][0]] ...]).
[2017-06-21 15:08:37,752][INFO ][cluster.service          ] [Triathlon] removed {{graylog-4e500f0d-eb65-45b1-9225-9452b329d6e4}{Ss44szpESsu1yYZQYrLSIw}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false},}, reason: zen-disco-node-left({graylog-4e500f0d-eb65-45b1-9225-9452b329d6e4}{Ss44szpESsu1yYZQYrLSIw}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false}), reason(left)
[2017-06-21 15:09:00,965][INFO ][cluster.service          ] [Triathlon] added {{graylog-4e500f0d-eb65-45b1-9225-9452b329d6e4}{JoimL-WjSICx0DZOJ68vuw}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false},}, reason: zen-disco-join(join from node[{graylog-4e500f0d-eb65-45b1-9225-9452b329d6e4}{JoimL-WjSICx0DZOJ68vuw}{127.0.0.1}{127.0.0.1:9350}{client=true, data=false, master=false}])

Any thoughts on where to check next? I also get the same error in the server if I go to streams to view or search for messages, doesn’t matter what timescale I put on it. When I look at the input node it shows 1 connection and indicates messages have been received, as does the server I have sending them (Windows server via nxlog)

Install details are:
Running CentOS 7 on a VM
6GB RAM (Heap set to 3GB)
Graylog 2.2.3
Elasticsearch 2.4.5
Mongo DB 3.4.5

Many thanks for any help or pointers!
gR


Unable to view messages on new install (Graylog 2.3 on Centos7)
(Jan Doberstein) #2

did you check your journal? Are any messages in Elasticsearch? Is Graylog able to send messages to Elasticsearch?


#3

Hi Jan, thanks for the reply. No errors in journalctl that I can see. Just setting this up now so if I’ve done this wrong please let me know but when I check curl -XGET http://localhost:9200/_stats/docs,store Elasticsearch is showing 1000s of messages so I’m assuming its Graylog putting them in there as nothing else would be configured to do so

{"_shards":{"total":1,"successful":1,"failed":0},"_all":{"primaries":{"docs":{"count":28720,"deleted":0},"store":{"size_in_bytes":15615501,"throttle_time_in_millis":0}},"total":{"docs":{"count":28720,"deleted":0},"store":{"size_in_bytes":15615501,"throttle_time_in_millis":0}}},"indices":{"graylog_0":{"primaries":{"docs":{"count":28720,"deleted":0},"store":{"size_in_bytes":15615501,"throttle_time_in_millis":0}},"total":{"docs":{"count":28720,"deleted":0},"store":{"size_in_bytes":15615501,"throttle_time_in_millis":0}}}}}

Thanks for the tips, appreciate any more pointers you could provide,
gR


(Jan Doberstein) #4

@gerryrigney you should check your graylog server.log and your elasticsearch log. Where to find depends on the setup but defaults are documented here.

Did System > Overview in the Graylog Webinterface gives you an idea what the problem is?


#5

No obvious errors in the logs with the exception of the:
[SearchResource] Unable to execute search: all shards failed
in the graylog.log everytime I do a search or go to view messages. Everything is green in the overview (see attached)

Thanks
gR


#6

Just to note, clean install using the guide: http://docs.graylog.org/en/2.3/pages/installation/os/centos.html and messages are now showing up.
Thanks


(santhosh) #7

Hello,

I too face the same issue.

Search area displays “loading” - unable to fetch any logs.

But I’m sure that receiving logs from sources - please find the below reference screenshot

Note: All Time stamp are same.

regards,


(system) #8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.