1. Describe your incident:
setup a minimun gralog6.x Cluster
2. Describe your environment:
-
OS Information:
redhat9.1 x86_64 -
Package Version:
6.1.3 -
Service logs, configurations, and environment variables:
3. What steps have you already taken to try and solve the problem?
i’ve downloaded the rpm :
graylog-datanode-6.1.3-1.x86_64.rpm
graylog-server-6.1.3-1.x86_64.rpm
mongodb-org-server-7.0.15-1.el9.x86_64.rpm
then i installed them all with rpm,
then i configured the ‘password_secret’
‘root_password_sha2’ in /etc/graylog/server/server.conf datanode.conf 、/etc/graylog/datanode/datanode.conf ，
then i started the datanode and server:
systemctl start graylog-server.service
systemctl start graylog-datanode.service
and visit the web interface : http ://192.168.1.76 and setup the certificate and restart it.
it works fine, in single node.
but , now i just want to setup a minimun gralog6.x Cluster .
i have two node:
192.168.1.73 , with hostname is localhost
192.168.1.76 , with hostname is graylogdatanode2
i install the same graylog-datanode、graylog-server rpm in another node with ip: 192.168.1.73. and i configured the same ‘password_secret’ ‘root_password_sha2’
and i set is_leader = false .
remove the certificates by:
rm -rf /var/lib/graylog-datanode/opensearch/config/*
and i removed all data in mogondb in collection graylog.
then i started the two datanode, and the graylog-server:
and then i reconfig the certificates via the server’s web interface : http://192.168.1.76
but the datanode cannt startup . as errors shown datanode.log:
2024-12-12T17:55:32.261+08:00 ERROR [NodeMetricsCollector] Error retrieving node stats for node graylogdatanode2
org.graylog.shaded.opensearch2.org.opensearch.client.ResponseException: method [GET], host [https://graylogdatanode2:9200], URI [_nodes/graylogdatanode2/stats], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -1212416b"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -1212416b","suppressed":[{"type":"illegal_state_exception","reason":"Failed to close the XContentBuilder","caused_by":{"type":"i_o_exception","reason":"Unclosed object or array found"}}]},"status":400}
at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.convertResponse(RestClient.java:376) ~[graylog-storage-opensearch2-6.1.3.jar:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:346) ~[graylog-storage-opensearch2-6.1.3.jar:?]
at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:321) ~[graylog-storage-opensearch2-6.1.3.jar:?]
at org.graylog.datanode.metrics.NodeMetricsCollector.getNodeContextFromRequest(NodeMetricsCollector.java:72) ~[graylog-datanode.jar:?]
at org.graylog.datanode.metrics.NodeMetricsCollector.getNodeMetrics(NodeMetricsCollector.java:53) ~[graylog-datanode.jar:?]
at org.graylog.datanode.periodicals.MetricsCollector.lambda$doRun$0(MetricsCollector.java:118) ~[graylog-datanode.jar:?]
at java.base/java.util.Optional.ifPresent(Unknown Source) [?:?]
at org.graylog.datanode.periodicals.MetricsCollector.doRun(MetricsCollector.java:109) [graylog-datanode.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99) [graylog2-server-6.1.3.jar:?]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
and the server also cannt startup , encountering errors:
ERROR [ServerBootstrap] Exception while running migrations
com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'Write operation error on server localhost:27017. Write error: WriteError{code=11000, message='E11000 duplicate key error collection: graylog.index_sets index: index_prefix_1 dup key: { index_prefix: "graylog" }', details={}}.'
at org.mongojack.JacksonDBCollection.possiblyAsDuplicateKeyError(JacksonDBCollection.java:386) ~[graylog.jar:?]
at org.mongojack.JacksonDBCollection.doSave(JacksonDBCollection.java:173) ~[graylog.jar:?]
at org.mongojack.JacksonDBCollection.save(JacksonDBCollection.java:155) ~[graylog.jar:?]
at org.mongojack.JacksonDBCollection.save(JacksonDBCollection.java:151) ~[graylog.jar:?]
at org.graylog2.indexer.indexset.MongoIndexSetService.save(MongoIndexSetService.java:177) ~[graylog.jar:?]
at org.graylog2.migrations.V20161116172100_DefaultIndexSetMigration.upgrade(V20161116172100_DefaultIndexSetMigration.java:87) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.lambda$runMigrations$3(ServerBootstrap.java:412) ~[graylog.jar:?]
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source) ~[?:?]
at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source) ~[?:?]
at com.google.common.collect.CollectSpliterators$1WithCharacteristics.lambda$forEachRemaining$1(CollectSpliterators.java:72) ~[graylog.jar:?]
at java.base/java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Unknown Source) ~[?:?]
at com.google.common.collect.CollectSpliterators$1WithCharacteristics.forEachRemaining(CollectSpliterators.java:72) ~[graylog.jar:?]
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) ~[?:?]
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source) ~[?:?]
at org.graylog2.bootstrap.ServerBootstrap.runMigrations(ServerBootstrap.java:409) ~[graylog.jar:?]
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:335) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:358) [graylog.jar:?]
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:270) [graylog.jar:?]
at org.graylog2.bootstrap.Main.main(Main.java:55) [graylog.jar:?]
4. How can the community help?
I just dont know how to setup a gralog6.x Cluster , the official document didnt say it clear either in Multi-Node Setup
I searched all the internet ,and find some old article, but there seems no one had ever shared how to setup a gralog6.x Cluster.
can anyone give me the step by step guidances to set up gralog6.x Cluster ?
How did the opensearch embebed in graylog-datanode, how graylog-datanode controls opensearch ?
Is that i have to setup a real opensearch Cluster before setup a multinode graylog-datanode Cluster ?
Is that graylog-datanode need to be a Cluster or just the graylog-server need to be a Cluster ?
What are the roles in the Cluster architecture?
God bless you.