So I have a graylog cluster with 2 nodes. the public load balancer has the 2 nodes attached.
At first on node A was the master and all the inputs were working ok. Now I switched node B to be master (chane configuration in both graylog.conf - A master:false B master:true and I see now B marked with a star as the master). But If I create a new input and I choose node B I´m getting this:
Is like successfully created but is not running. I check the logs in graylog B node And I´m getting:
root@GrayLog-B:/var/log/graylog-server# tail -f server.log
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[graylog.jar:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[graylog.jar:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.executeRequest(JestHttpClient.java:151) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:77) ~[graylog.jar:?]
at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:64) ~[graylog.jar:?]
at org.graylog2.indexer.indices.Indices.aliasExists(Indices.java:292) ~[graylog.jar:?]
... 13 more
2019-08-28T15:01:05.622-04:00 ERROR [IndexFieldTypePoller] Couldn't get mapping for index <graylog_2>: Read timed out.
Any idea of error? Am I missing some other configuration in Node B so all inputs starts going through it?
I Have some other indeces created that were created when node A was master. Should I remove them and create them again while now B node is master?.
yes they are, in fact if both mongod services are not running, graylog also not working at all and you receive message that node cluster is unable to reach the other node mongo service
how can I configure that? global or local inputs. The way I have it now, when yo create a new one, it gives you the option to choose between node A or node B. Whn I configured some inputs first on node A (when A was master). For example from severX using fluentd to send logs to graylog I configured from serv like this:
and then I created an input, and I would choose Node A(10.0.6.35), port 12225 and it would start to bring all messages from ServerX. Now if I do exact the same and instead of host 10.0.6.35, I put in Server X 10.0.6.36 (Node B which now is master) It brings the error message that I first post. Not running. that´s why I think I am missing some other configuration appart from putting Node B as master.
Graylog’s documentation doesn’t go into relica sets much, but MongoDBs documentation does and I would just have you check the mongodb documentation. While you can build a replica set for MongoDB with 2 nodes, it’s recommend to have 3 because you need to have an odd number for primary voting. So double check your mongoDB replica set config. If there is an issue with MongoDB, I don’t believe Graylog will start as it holds all the configuration information.
local vs global inputs are selected in the input creation/configuration screen.
By default, an input is local and you need to select a node which will have that input exist only on that node.
Checking the global box removes the node selection option so that any of the nodes can be used for this input.
and what about output on server sending logs to graylog? Should this be reconfigured? Right now is just pointing to nodeB. If I set global input shouldn´t that be for either nodeA or B?
#Match to Graylog
<match tomcat.*.log>
type copy
buffer_type memory
buffer_chunk_limit 256m
buffer_queue_limit 128
flush_interval 1s
disable_retry_limit false
retry_limit 17
retry_wait 1s
type gelf
host 10.0.6.35
port 12225
yes… you’ll need to send it to the node you are ingesting it on. If you want to set up a global input for multiple servers, consider putting a load balancer in front of the 2 GL servers. The IP address in your source is then just a single IP address and the LB can handle distributing the actual traffic to the backend GL nodes.
I already have a public IP in a load balancer for this cluster. I gues I need to configure this public IP in the source server in fluendtd and configure the global input
yes I have it, in fact if one of mongod service is not running in one node, graylog is unaccesible. Her is what I have configured in /etc/graylog/server/server.conf in both nodes
and of course configure the LB to send the traffic onto the GL nodes. I am still curious if you are running a 2 node MongoDB replica set.