Inquiry about the Graylog Web Interface not working

Hi @all,

The Graylog log system has a search 500 error and has restarted the system.
Demons such as graylog-server are running after the restart. but the Graylog web interface does not work. As a result of examining the log, the contents below are written, so can you tell me what the problem is? please see below.

2022-04-19T18:09:51.282+09:00 WARN [KafkaJournal] Journal utilization (100.0%) has gone over 95%.
2022-04-19T18:09:51.290+09:00 ERROR [Cluster] Couldn’t read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)

Hello && Welcome

If your journal is full ( 5GB is default) I would look at elasticsearch. Seams that elasticsearch is not indexing those logs , hence journal is filling up.
Here are a couple options you try.

ES Health.

curl -XGET http://localhost:9200/_cluster/health?pretty

ES Shard Info, If something is wrong ( ES is in RED) this should give you an idea what to look for.

curl -XGET http://localhost:9200/_cluster/allocation/explain?pretty

That states Graylog is unable to connect to Elasticsearch using http://127.0.0.1:9200/

It could be Elasticsearch crashed, stopped , etc…

Best idea is to insure Elasticsearch is functioning correctly then start Graylog service.

thanks for your prompt reply and answering my question. As a result of examining the Easticsearch log, the contents below are written, so can you tell me what the problem is?
Both gray log servers and Elasticsearch are active and running. However, the web interface still does not open. I’d appreciate your help.

2022-04-20T17:37:48,413][INFO ][o.e.p.PluginsService     ] [8atbN9f] loaded module [aggs-matrix-stats]
[2022-04-20T17:37:48,413][INFO ][o.e.p.PluginsService     ] [8atbN9f] loaded module [ingest-common]
[2022-04-20T17:37:48,414][INFO ][o.e.p.PluginsService     ] [8atbN9f] loaded module [lang-expression]
[2022-04-20T17:37:48,414][INFO ][o.e.p.PluginsService     ] [8atbN9f] loaded module [lang-groovy]
[2022-04-20T17:37:48,414][INFO ][o.e.p.PluginsService     ] [8atbN9f] loaded module [lang-mustache]
[2022-04-20T17:37:48,414][INFO ][o.e.p.PluginsService     ] [8atbN9f] loaded module [lang-painless]
[2022-04-20T17:37:48,414][INFO ][o.e.p.PluginsService     ] [8atbN9f] loaded module [parent-join]
[2022-04-20T17:37:48,414][INFO ][o.e.p.PluginsService     ] [8atbN9f] loaded module [percolator]
[2022-04-20T17:37:48,414][INFO ][o.e.p.PluginsService     ] [8atbN9f] loaded module [reindex]
[2022-04-20T17:37:48,414][INFO ][o.e.p.PluginsService     ] [8atbN9f] loaded module [transport-netty3]
[2022-04-20T17:37:48,414][INFO ][o.e.p.PluginsService     ] [8atbN9f] loaded module [transport-netty4]
[2022-04-20T17:37:48,415][INFO ][o.e.p.PluginsService     ] [8atbN9f] no plugins loaded
[2022-04-20T17:37:49,580][INFO ][o.e.d.DiscoveryModule    ] [8atbN9f] using discovery type [zen]
[2022-04-20T17:37:50,083][INFO ][o.e.n.Node               ] initialized
[2022-04-20T17:37:50,083][INFO ][o.e.n.Node               ] [8atbN9f] starting ...
[2022-04-20T17:38:00,248][INFO ][o.e.t.TransportService   ] [8atbN9f] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2022-04-20T17:38:03,315][INFO ][o.e.c.s.ClusterService   ] [8atbN9f] new_master {8atbN9f}{8atbN9f9SBmUqI6mo4lnhw}{h2uYPLwMR5iT2xa3Lj2X2w}{127.0.0.1}{127.0.0.1:9300}, reason: zen-di
[co-elected-as-master ([]] nodes joined)
[2022-04-20T17:38:03,337][INFO ][o.e.h.n.Netty4HttpServerTransport] [8atbN9f] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2022-04-20T17:38:03,337][INFO ][o.e.n.Node               ] [8atbN9f] started
[2022-04-20T17:38:03,693][INFO ][o.e.g.GatewayService     ] [8atbN9f] recovered [20] indices into cluster_state
[                       ]                                                                                                                                                            
[2022-04-20T17:38:06,101][INFO ][o.e.c.r.a.AllocationService] [8atbN9f] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_92][0], [graylog_92][3@

@JaeiL

First, I fixed you post so its easier to read. Hope you don’t mind.
Second, This is good news from that log clip. Looks like its try to fix the issue for you.

2022-04-20T17:38:06,101][INFO ][o.e.c.r.a.AllocationService] [8atbN9f] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[graylog_92][0], [graylog_92][3@

I would need to see how you configured your Graylog server ( i.e. Configuration file, etc…)

thanks for your prompt reply and answering my question.
There seems to be a problem with the Elasticsearch connection. It’s not open with 9000, 9200 ports. When viewed on the web, Nodes displays the status as unknown. The logs on the server.
I’d appreciate your help.

00 WARN [ProxiedResource] Unable to call https://xxx.xxx.xxx.xxx:9000/api/system on node java.net.SocketTimeoutException: timeout
ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.net.SocketTimeoutException: timeout
2022-04-21T19:42:14.330+09:00 ERROR [Cluster] Couldn’t read cluster health for indices [graylog_*] (Could not connect to http://127.0.0.1:9200)
2022-04-21T19:42:14.330+09:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2022-04-21T19:42:14.451+09:00 WARN [V20161130141500_DefaultStreamRecalcIndexRanges] Interrupted or timed out waiting for Elasticsearch cluster, checking again.

Something is wrong with your Configuration for HTTPS

Something is either wrong with your configuration or perhaps block the connection to Elasticsearch

As I stated before

Thank you for your attention to this matter. I will check the gray log and check the connection error and configuration in Elasticsearch in detail.
I will update here whatever my observations will be. Thank you so much.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.