Elasticsearch exception

Hello all,

I am new in graylog and after installing the graylog, I have faced this isse:

While retrieving data for this widget, the following error(s) occurred:

  • Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];].

Could someonehelp on how to fix this? I have researched online and the troubleshooting was to run the command: ```
curl -X PUT “localhost:9200/_all/_settings” -H ‘Content-Type: application/json’ -d’{ “index.blocks.read_only” : false } }’


but still no joy.


The service is running (elasticsearch and graylog), I can log in in the admin panel but only facing that issue.

![image|690x267](upload://i5GEhU8yVI7XFQIDQbtPtPOgZy5.png)

It would help if you posted more about your setup, any changes you might have made, why you thought the command you ran would solve the problem. At the bottom is a link to guide you to providing more information.

In the mean time, did you run out of space on your elasticsearch server? Have you checked the Graylog logs and the elasticsearch logs?

tail -f /var/log/graylog-server/server.log

(Elasticsearch is whatever you called your instance located in /var/log/elasticsearch… possibly graylog.log)

Don’t post the whole log, just parts that are relevant - and please use the forum tools such as </> to make your code/logs more readable. Also review below for help on giving good information when asking a question. :slight_smile:

This is the outpout

2021-11-08T15:06:04.982Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:06.584Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:06.980Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:07.310Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:08.582Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:08.984Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:10.583Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:10.984Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:12.314Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:12.581Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:13.985Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:14.584Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:15.977Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:16.584Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:17.315Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:17.985Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:18.589Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:19.997Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:20.586Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:21.987Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:22.320Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:22.586Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:23.989Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-08T15:06:24.587Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000

I have fixed that parted and believe because I have changed my data store paths.

Now in the nodes it gives the error mentioned above when clicking on the node and below the node name says system information is currently unavailable.

And in the log it says this

Error getting data

We had trouble fetching some data required to build this page, so here is a picture instead.

FetchError: There was an error fetching a resource: Internal Server Error. Additional information: Failed to connect to server/127.0.1.1:9000
Check your Graylog logs for more information.

The output… from what? It looks like the same lines repeated so we would be better served by just putting in one one and saying it repeats… Your follow up post doesn’t tell me much either - you fixed what part? you changes data store paths… for what mongoDB? Elaticsearch? If you did either of those did you adjust the Graylog configuration file to point to it? What was it that you were doing before the problem happened? What have you tried to do to fix it? Are your graylog logs different now that you fixed whatever part you fixed? Can you post your Graylog configuration … use this command for just the details:

cat /etc/graylog/server/server.conf | egrep -v "^\s*(#|$)"

I need you to be a little bit more diligent with giving me information

Apologises for the confusion.
The 1st issue ( While retrieving data for this widget, the following error(s) occurred:

  • Elasticsearch exception [type=cluster_block_exception, reason=blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];].) was fixed due the changes in the paths in the elasticsearch.yml file. HAS BEEN SOLVED

The 2nd issue is the info that displayes in the node:

There is 1 active node

servername/servername…

System information is currently unavailable.

When clicking in the server name it then displays

This is the output:
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = nXktrsRNh5pxabPK8iYYBmSaCTEe00IPZcyf1C88m7rnNLBGRG6IXqhxv9MvInRegNEA71J14hELq7jU75qKFe5sCNQYwBCpQy
root_password_sha2 = 54f423a7f5e7c619d07418254cebeff4a724eba56642cf06950cab98f581587171e0
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_bind_address = #########:9000
http_publish_uri = http://servername:9000
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

I don’t see anything in the graylog server.conf that tells Graylog where your elasticsearch_hosts are… that must be defined - look for this commented line.

#elasticsearch_hosts = http://node1:9200,http://user:password@node2:19200

I don’t know what version of Graylog you are on but you might have to define elasticsearch_version as well…

Adding on to what @tmacgbay suggested.

If this is a fresh install try just using this setting.

http_bind_address = ip_address:9000

Comment this out

http_publish_uri = http://servername:9000
#http_publish_uri = http://servername:9000

If your using your local IP Address (192.168.1.10) for your URL and not your loop back ( 127.0.0.1.) without a DNS server make sure your /etc/host file is correct.

If you make this changes that @tmacgbay and I suggest, don’t forget to restart Graylog service.

1 Like

Hello,

Thank you for your support.

I have to use DNS so that is why I had uncomment and edit the http_publish_uri.

And just to check in the elastic search_hosts do I have to add my ip addresss/domain name, the user and the password?

Eg: elasticsearch_hosts = http://myip:9200,http://user:password@node2:19200

The issue is still the same after restarting.

Error getting data

We had trouble fetching some data required to build this page, so here is a picture instead.

FetchError: There was an error fetching a resource: Internal Server Error. Additional information: Failed to connect to myserver/127.0.1.1:9000
Check your Graylog logs for more information.

And the graylog log does say the same:
2021-11-09T09:21:46.833Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-09T09:21:47.267Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/metrics/multiple on node : Failed to connect to servername/127.0.1.1:9000
2021-11-09T09:21:47.271Z WARN [ProxiedResource] Unable to call http://servername:9000/api/system/jobs on node : Failed to connect to servername/127.0.1.1:9000

And this is my elasticsearch.yml file

image

HI guys,

Thank you very much for your support.

I did not know that for using the dns name I only had to update in the hostname file.

I have comment the #http_publish_uri and I can see the messages with no problem now. Thank you for you support

1 Like

Nice glad I could help :slight_smile:

EDIT: I just noticed your configuration on Elasticsearch. Some settings don.t look right. Data path looks wrong in your configuration file.

Here is mine Elasticsearch version 7.10

cluster.name: graylog
path.data: /var/lib/elasticsearch 
path.logs: /var/log/elasticsearch
network.host: 10.10.10.10
http.port: 9200
action.auto_create_index: false

It should be set like this unless you secured your elasticsearch. If its a default configuration no need to use a username or password . The following setting in your Graylog config file should be good. Take note this is version 4.2 and I’m not sure what version you have.

elasticsearch_hosts = http://10.50.1.40:9200

Remember to restart your service after adjusting your config files.

I have changed the path to store the logs so that is why it looks different.

I understand, never seen Elasticsearch home directory in the Log files directory :slight_smile:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.