Unable to get messages in graylog web GUI

Graylog Central (peer support)
1

Hi,
I have successfully configured graylog server. and unable to get the messages in graylog web GUI. I am getting error as
"Could not execute search
There was an error executing your search. Please check your Graylog server logs for more information.

Error Message:
blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];: cannot GET http://192.168.2.147:12900/search/universal/relative?query=gl2_source_input%3A5af29f7d91fbb43a01dca396&range=28800&limit=150&sort=timestamp%3Adesc (500)
Search status code:
500"

and in client machine, i have done rsyslog configuration too. I.e in client machine, i have done the following things.

  1. Opened #vi /etc/rsyslog.d/90-graylog2.conf
$template GRAYLOGRFC5424,"%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msg%\n"
*.* @<GraylogServerIP>:5140;GRAYLOGRFC5424

  1. opened #vi /etc/rsyslog.conf

                    $ModLoad imudp
                $UDPServerRun 514
                $ModLoad imtcp
                $InputTCPServerRun 514
                *.* @ <graylogServerIP>:514

and then restarted rsyslog service.

Now getting error as said above and unable to get messages. Please suggest me if i went wrong.

2

What’s in the logs of your Graylog and Elasticsearch nodes?
http://docs.graylog.org/en/2.4/pages/configuration/file_location.html

Please also refer to the Syslog configuration guide for Graylog for information about setting up rsyslog:

3

2018-05-09T12:34:24.784+05:30 INFO [CmdLineTool] Loaded plugin: Elastic Beats Input 2.2.3 [org.graylog.plugins.beats.BeatsInputPlugin]
2018-05-09T12:34:24.785+05:30 INFO [CmdLineTool] Loaded plugin: Collector 2.2.3 [org.graylog.plugins.collector.CollectorPlugin]
2018-05-09T12:34:24.786+05:30 INFO [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.2.3 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2018-05-09T12:34:24.786+05:30 INFO [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.2.3 [org.graylog.plugins.map.MapWidgetPlugin]
2018-05-09T12:34:24.792+05:30 INFO [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.2.3 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2018-05-09T12:34:24.792+05:30 INFO [CmdLineTool] Loaded plugin: Anonymous Usage Statistics 2.2.3 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2018-05-09T12:34:25.098+05:30 INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=rpm
2018-05-09T12:34:25.283+05:30 INFO [Version] HV000001: Hibernate Validator null
2018-05-09T12:34:26.963+05:30 INFO [InputBufferImpl] Message journal is enabled.
2018-05-09T12:34:27.021+05:30 INFO [NodeId] Node ID: 7eb22e66-e7b8-4427-abf9-6ecff00d5964
2018-05-09T12:34:27.162+05:30 INFO [LogManager] Loading logs.
2018-05-09T12:34:27.166+05:30 INFO [LogManager] Logs loading complete.
2018-05-09T12:34:27.232+05:30 INFO [LogManager] Created log for partition [messagejournal,0] in /var/lib/graylog-server/journal with properties {file.delete.delay.ms -> 60000, compact -> false, max.message.bytes -> 104857600, min.insync.replicas -> 1, segment.jitter.ms -> 0, index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5, unclean.leader.election.enable -> true, retention.bytes -> 5368709120, delete.retention.ms -> 86400000, flush.ms -> 60000, segment.bytes -> 104857600, segment.ms -> 3600000, retention.ms -> 43200000, flush.messages -> 1000000, segment.index.bytes -> 1048576}.
2018-05-09T12:34:27.232+05:30 INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2018-05-09T12:34:27.243+05:30 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy , running 2 parallel message handlers.
2018-05-09T12:34:27.255+05:30 INFO [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout=‘30000 ms’, maxWaitQueueSize=5000}
2018-05-09T12:34:27.291+05:30 INFO [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2018-05-09T12:34:27.662+05:30 INFO [connection] Opened connection [connectionId{localValue:1, serverValue:1}] to localhost:27017
2018-05-09T12:34:27.664+05:30 INFO [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 20]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=664420}
2018-05-09T12:34:27.688+05:30 INFO [connection] Opened connection [connectionId{localValue:2, serverValue:2}] to localhost:27017
2018-05-09T12:34:28.332+05:30 INFO [node] [graylog-7eb22e66-e7b8-4427-abf9-6ecff00d5964] version[2.4.4], pid[14849], build[fcbb46d/2017-01-03T11:33:16Z]
2018-05-09T12:34:28.332+05:30 INFO [node] [graylog-7eb22e66-e7b8-4427-abf9-6ecff00d5964] initializing …
2018-05-09T12:34:28.343+05:30 INFO [plugins] [graylog-7eb22e66-e7b8-4427-abf9-6ecff00d5964] modules [], plugins [graylog-monitor], sites []
2018-05-09T12:34:45.725+05:30 INFO [node] [graylog-7eb22e66-e7b8-4427-abf9-6ecff00d5964] initialized
2018-05-09T12:34:46.107+05:30 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy .
2018-05-09T12:34:48.709+05:30 INFO [RulesEngineProvider] No static rules file loaded.
2018-05-09T12:34:49.070+05:30 WARN [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb

4

Below are elastic search Logs

[2018-05-09 12:30:00,190][INFO ][node ] [Elathan] version[2.4.6], pid[14543], build[5376dca/2017-07-18T12:17:44Z]
[2018-05-09 12:30:00,191][INFO ][node ] [Elathan] initializing …
[2018-05-09 12:30:00,465][INFO ][plugins ] [Elathan] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2018-05-09 12:30:00,526][INFO ][env ] [Elathan] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [42.5gb], net total_space [46.9gb], spins? [unknown], types [rootfs]
[2018-05-09 12:30:00,526][INFO ][env ] [Elathan] heap size [990.7mb], compressed ordinary object pointers [true]
[2018-05-09 12:30:01,544][INFO ][node ] [Elathan] initialized
[2018-05-09 12:30:01,546][INFO ][node ] [Elathan] starting …
[2018-05-09 12:30:01,628][INFO ][transport ] [Elathan] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-05-09 12:30:01,633][INFO ][discovery ] [Elathan] graylog/Fs4GZz1SQhOErFTU_0z2-A
[2018-05-09 12:30:04,656][INFO ][cluster.service ] [Elathan] new_master {Elathan}{Fs4GZz1SQhOErFTU_0z2-A}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2018-05-09 12:30:04,664][INFO ][http ] [Elathan] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-05-09 12:30:04,664][INFO ][node ] [Elathan] started
[2018-05-09 12:30:04,678][INFO ][gateway ] [Elathan] recovered [0] indices into cluster_state
[2018-05-09 12:58:57,437][INFO ][node ] [Elathan] stopping …
[2018-05-09 12:59:00,046][INFO ][node ] [Elathan] stopped
[2018-05-09 12:59:00,047][INFO ][node ] [Elathan] closing …
[2018-05-09 12:59:01,327][INFO ][node ] [Elathan] closed
[2018-05-09 12:59:04,181][INFO ][node ] [Centennial] version[2.4.6], pid[15602], build[5376dca/2017-07-18T12:17:44Z]
[2018-05-09 12:59:04,181][INFO ][node ] [Centennial] initializing …
[2018-05-09 12:59:05,253][INFO ][plugins ] [Centennial] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2018-05-09 12:59:05,362][INFO ][env ] [Centennial] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [42.3gb], net total_space [46.9gb], spins? [unknown], types [rootfs]
[2018-05-09 12:59:05,363][INFO ][env ] [Centennial] heap size [990.7mb], compressed ordinary object pointers [true]
[2018-05-09 12:59:20,611][INFO ][node ] [Centennial] initialized
[2018-05-09 12:59:20,618][INFO ][node ] [Centennial] starting …
[2018-05-09 12:59:20,953][INFO ][transport ] [Centennial] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-05-09 12:59:21,004][INFO ][discovery ] [Centennial] graylog/lT0si95HQ7yVXoWrbXfXGA
[2018-05-09 12:59:24,048][INFO ][cluster.service ] [Centennial] new_master {Centennial}{lT0si95HQ7yVXoWrbXfXGA}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2018-05-09 12:59:24,059][INFO ][http ] [Centennial] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-05-09 12:59:24,059][INFO ][node ] [Centennial] started
[2018-05-09 12:59:24,082][INFO ][gateway ] [Centennial] recovered [0] indices into cluster_state
[2018-05-09 14:30:56,540][INFO ][node ] [Centennial] stopping …
[2018-05-09 14:30:59,875][INFO ][node ] [Centennial] stopped
[2018-05-09 14:30:59,875][INFO ][node ] [Centennial] closing …
[2018-05-09 14:31:00,368][INFO ][node ] [Centennial] closed
[2018-05-09 14:31:37,930][INFO ][node ] [Capricorn] version[2.4.6], pid[1147], build[5376dca/2017-07-18T12:17:44Z]

5

Are these the complete logs of Graylog and Elasticsearch?

Unless there are more messages in the Elasticsearch logs, it looks like the ES node isn’t running.

6

Those are all Logs. seems issue is with elastic search cluster. it is still in “updating” status in the graylog web GUI.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.