Adding a node to a single cluster

Hi all,

Setup is a single host. Version are:

Host - Ubuntu 18.04
Graylog - 3.0.0-12
Java - 1.8.0_191
Elasticsearch - 6.6.2"
MongoDB - 4.0.6
Apache - 2.4.29

I’m not able to see and data in the sources tab and under overview It states " There were 204,847 failed indexing attempts in the last 24 hours." Clicking on more information the error message is: “{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}”

I don’t understate what is wrong from the error message, any one come run into this? and how to resolve it?

Thanks

Please try to read the error message again.
And again.
The I suggest to read graylog docs FAQ. There are a lot of useful information about graylog’s components.
After I can suggest a fast search ín the community for index failure.
And after check the System/ovetwiev menu, and your elastic cluster status. I think it is red. Or you have problem with deflector alias.

Elasticsearch (6) will make your index “read-only” if it hits it low-watermark.

That means, not enough space on disk! Check your Elasticsearch Logs

Thanks both,

Yea I have checked the ES logs. I only have INFO logs for a second node not connecting (currently setting up a second node )

[2019-03-27T09:02:06,057][INFO ][o.e.d.z.ZenDiscovery ] [grayone] failed to send join request to master [{crow}{KQtVL-x1TbWLGcvDiGyXLA}{4_Ae58sDRty8Dwa6i-Zouw}{172.24.228.226}{172.24.228.226:9300}{ml.machine_memory=67529670656, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[crow][172.24.228.226:9300][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: RemoteTransportException[[grayone][172.24.228.161:9300][internal:discovery/zen/join/validate]]; nested: IllegalArgumentException[Unknown NamedWriteable [org.elasticsearch.cluster.metadata.MetaData$Custom][licenses]]; ]

But this wouldn’t effect the main node?

Yea checked the disk space and it has plenty there and I have set cluster.routing.allocation.disk.threshold_enabled: false in the ES conf

Filesystem      Size  Used Avail Use% Mounted on
udev             32G     0   32G   0% /dev
tmpfs           6.3G  1.4M  6.3G   1% /run
/dev/sda2       246G   94G  140G  41% /
tmpfs            32G     0   32G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            32G     0   32G   0% /sys/fs/cgroup
tmpfs            32G     0   32G   0% /run/shm
/dev/loop0       91M   91M     0 100% /snap/core/6405
/dev/loop1       87M   87M     0 100% /snap/core/4486
/dev/loop2       12M   12M     0 100% /snap/lnav/116
/dev/loop3       92M   92M     0 100% /snap/core/6531
tmpfs           6.3G     0  6.3G   0% /run/user/1000

Checking ES health it returns

{
“error” : {
“root_cause” : [
{
“type” : “master_not_discovered_exception”,
“reason” : null
}
],
“type” : “master_not_discovered_exception”,
“reason” : null
},
“status” : 503
}

Which is odd because

is_master = true

is set in the graylog.conf and just a basic conf for ES (below)

cluster.name: graylog
action.auto_create_index: false

cluster.routing.allocation.disk.threshold_enabled: false
network.host: 172.24.228.161
node.name: grayone

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

discovery.zen.ping.unicast.hosts: [“172.24.228.161”,“172.24.228.226”]

I have RTFM, but its not specific and a bit vague with the ES set up. Not sure where to go from here?

You missed something, because you try to debug elastic, and copy a graylog config option.
I suggest read graylog faq, to make components clear.

After read again what you wrote.

I would like to share my best friend’s resources with you.

There are a lot of information about this problem.

You wrote about single node, but your elasticsaerch told two different IP addresses. Maybe you missconfigure something…

Not debugged anything, that was the output on the gray log node

curl -XGET ‘localhost:9200/_cluster/health?pretty’

If google had my answer, I would be asking for help here :wink:

Yea so I have my single graylog node. Now I’m trying to add a second node as per https://www.graylog.org/post/back-to-basics-from-single-server-to-graylog-cluster

The second IP is the second node. The error for

“{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}” Has been fixed. Ended up needing to remove the data from ES, logs now coming in

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.