Adding a node to a single cluster

Graylog Central (peer support)
1

Hi all,

Setup is a single host. Version are:

Host - Ubuntu 18.04
Graylog - 3.0.0-12
Java - 1.8.0_191
Elasticsearch - 6.6.2"
MongoDB - 4.0.6
Apache - 2.4.29

I’m not able to see and data in the sources tab and under overview It states " There were 204,847 failed indexing attempts in the last 24 hours." Clicking on more information the error message is: “{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}”

I don’t understate what is wrong from the error message, any one come run into this? and how to resolve it?

Thanks

2

Please try to read the error message again.
And again.
The I suggest to read graylog docs FAQ. There are a lot of useful information about graylog’s components.
After I can suggest a fast search ín the community for index failure.
And after check the System/ovetwiev menu, and your elastic cluster status. I think it is red. Or you have problem with deflector alias.

3

Elasticsearch (6) will make your index “read-only” if it hits it low-watermark.

That means, not enough space on disk! Check your Elasticsearch Logs

4

Thanks both,

Yea I have checked the ES logs. I only have INFO logs for a second node not connecting (currently setting up a second node )

[2019-03-27T09:02:06,057][INFO ][o.e.d.z.ZenDiscovery ] [grayone] failed to send join request to master [{crow}{KQtVL-x1TbWLGcvDiGyXLA}{4_Ae58sDRty8Dwa6i-Zouw}{172.24.228.226}{172.24.228.226:9300}{ml.machine_memory=67529670656, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[crow][172.24.228.226:9300][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: RemoteTransportException[[grayone][172.24.228.161:9300][internal:discovery/zen/join/validate]]; nested: IllegalArgumentException[Unknown NamedWriteable [org.elasticsearch.cluster.metadata.MetaData$Custom][licenses]]; ]

But this wouldn’t effect the main node?

Yea checked the disk space and it has plenty there and I have set cluster.routing.allocation.disk.threshold_enabled: false in the ES conf

Filesystem      Size  Used Avail Use% Mounted on
udev             32G     0   32G   0% /dev
tmpfs           6.3G  1.4M  6.3G   1% /run
/dev/sda2       246G   94G  140G  41% /
tmpfs            32G     0   32G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            32G     0   32G   0% /sys/fs/cgroup
tmpfs            32G     0   32G   0% /run/shm
/dev/loop0       91M   91M     0 100% /snap/core/6405
/dev/loop1       87M   87M     0 100% /snap/core/4486
/dev/loop2       12M   12M     0 100% /snap/lnav/116
/dev/loop3       92M   92M     0 100% /snap/core/6531
tmpfs           6.3G     0  6.3G   0% /run/user/1000

Checking ES health it returns

{
“error” : {
“root_cause” : [
{
“type” : “master_not_discovered_exception”,
“reason” : null
}
],
“type” : “master_not_discovered_exception”,
“reason” : null
},
“status” : 503
}

Which is odd because

is_master = true

is set in the graylog.conf and just a basic conf for ES (below)

cluster.name: graylog
action.auto_create_index: false

cluster.routing.allocation.disk.threshold_enabled: false
network.host: 172.24.228.161
node.name: grayone

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

discovery.zen.ping.unicast.hosts: [“172.24.228.161”,“172.24.228.226”]

I have RTFM, but its not specific and a bit vague with the ES set up. Not sure where to go from here?

5

You missed something, because you try to debug elastic, and copy a graylog config option.
I suggest read graylog faq, to make components clear.

After read again what you wrote.

I would like to share my best friend’s resources with you.

There are a lot of information about this problem.

You wrote about single node, but your elasticsaerch told two different IP addresses. Maybe you missconfigure something…

6

Not debugged anything, that was the output on the gray log node

curl -XGET ‘localhost:9200/_cluster/health?pretty’

If google had my answer, I would be asking for help here :wink:

Yea so I have my single graylog node. Now I’m trying to add a second node as per https://www.graylog.org/post/back-to-basics-from-single-server-to-graylog-cluster

The second IP is the second node. The error for

“{“type”:“cluster_block_exception”,“reason”:“blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];”}” Has been fixed. Ended up needing to remove the data from ES, logs now coming in

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.