How to setup a minimum gralog6.x Cluster step by step

1. Describe your incident:
setup a minimun gralog6.x Cluster

2. Describe your environment:

  • OS Information:
    redhat9.1 x86_64

  • Package Version:
    6.1.3

  • Service logs, configurations, and environment variables:

3. What steps have you already taken to try and solve the problem?

i’ve downloaded the rpm :
graylog-datanode-6.1.3-1.x86_64.rpm
graylog-server-6.1.3-1.x86_64.rpm
mongodb-org-server-7.0.15-1.el9.x86_64.rpm

then i installed them all with rpm,

then i configured the ‘password_secret’
‘root_password_sha2’ in /etc/graylog/server/server.conf datanode.conf 、/etc/graylog/datanode/datanode.conf ,

then i started the datanode and server:
systemctl start graylog-server.service
systemctl start graylog-datanode.service

and visit the web interface : http ://192.168.1.76 and setup the certificate and restart it.

it works fine, in single node.

but , now i just want to setup a minimun gralog6.x Cluster .
i have two node:
192.168.1.73 , with hostname is localhost
192.168.1.76 , with hostname is graylogdatanode2

i install the same graylog-datanode、graylog-server rpm in another node with ip: 192.168.1.73. and i configured the same ‘password_secret’ ‘root_password_sha2’

and i set is_leader = false .

remove the certificates by:
rm -rf /var/lib/graylog-datanode/opensearch/config/*

and i removed all data in mogondb in collection graylog.
then i started the two datanode, and the graylog-server:

and then i reconfig the certificates via the server’s web interface : http://192.168.1.76

but the datanode cannt startup . as errors shown datanode.log:

2024-12-12T17:55:32.261+08:00 ERROR [NodeMetricsCollector] Error retrieving node stats for node graylogdatanode2
org.graylog.shaded.opensearch2.org.opensearch.client.ResponseException: method [GET], host [https://graylogdatanode2:9200], URI [_nodes/graylogdatanode2/stats], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -1212416b"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -1212416b","suppressed":[{"type":"illegal_state_exception","reason":"Failed to close the XContentBuilder","caused_by":{"type":"i_o_exception","reason":"Unclosed object or array found"}}]},"status":400}
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.convertResponse(RestClient.java:376) ~[graylog-storage-opensearch2-6.1.3.jar:?]
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:346) ~[graylog-storage-opensearch2-6.1.3.jar:?]
	at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:321) ~[graylog-storage-opensearch2-6.1.3.jar:?]
	at org.graylog.datanode.metrics.NodeMetricsCollector.getNodeContextFromRequest(NodeMetricsCollector.java:72) ~[graylog-datanode.jar:?]
	at org.graylog.datanode.metrics.NodeMetricsCollector.getNodeMetrics(NodeMetricsCollector.java:53) ~[graylog-datanode.jar:?]
	at org.graylog.datanode.periodicals.MetricsCollector.lambda$doRun$0(MetricsCollector.java:118) ~[graylog-datanode.jar:?]
	at java.base/java.util.Optional.ifPresent(Unknown Source) [?:?]
	at org.graylog.datanode.periodicals.MetricsCollector.doRun(MetricsCollector.java:109) [graylog-datanode.jar:?]
	at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99) [graylog2-server-6.1.3.jar:?]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.base/java.lang.Thread.run(Unknown Source) [?:?]

and the server also cannt startup , encountering errors:

ERROR [ServerBootstrap] Exception while running migrations
com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'Write operation error on server localhost:27017. Write error: WriteError{code=11000, message='E11000 duplicate key error collection: graylog.index_sets index: index_prefix_1 dup key: { index_prefix: "graylog" }', details={}}.'
	at org.mongojack.JacksonDBCollection.possiblyAsDuplicateKeyError(JacksonDBCollection.java:386) ~[graylog.jar:?]
	at org.mongojack.JacksonDBCollection.doSave(JacksonDBCollection.java:173) ~[graylog.jar:?]
	at org.mongojack.JacksonDBCollection.save(JacksonDBCollection.java:155) ~[graylog.jar:?]
	at org.mongojack.JacksonDBCollection.save(JacksonDBCollection.java:151) ~[graylog.jar:?]
	at org.graylog2.indexer.indexset.MongoIndexSetService.save(MongoIndexSetService.java:177) ~[graylog.jar:?]
	at org.graylog2.migrations.V20161116172100_DefaultIndexSetMigration.upgrade(V20161116172100_DefaultIndexSetMigration.java:87) ~[graylog.jar:?]
	at org.graylog2.bootstrap.ServerBootstrap.lambda$runMigrations$3(ServerBootstrap.java:412) ~[graylog.jar:?]
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source) ~[?:?]
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source) ~[?:?]
	at com.google.common.collect.CollectSpliterators$1WithCharacteristics.lambda$forEachRemaining$1(CollectSpliterators.java:72) ~[graylog.jar:?]
	at java.base/java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Unknown Source) ~[?:?]
	at com.google.common.collect.CollectSpliterators$1WithCharacteristics.forEachRemaining(CollectSpliterators.java:72) ~[graylog.jar:?]
	at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) ~[?:?]
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source) ~[?:?]
	at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
	at java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source) ~[?:?]
	at org.graylog2.bootstrap.ServerBootstrap.runMigrations(ServerBootstrap.java:409) ~[graylog.jar:?]
	at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:335) [graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.doRun(CmdLineTool.java:358) [graylog.jar:?]
	at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:270) [graylog.jar:?]
	at org.graylog2.bootstrap.Main.main(Main.java:55) [graylog.jar:?]

4. How can the community help?
I just dont know how to setup a gralog6.x Cluster , the official document didnt say it clear either in Multi-Node Setup

I searched all the internet ,and find some old article, but there seems no one had ever shared how to setup a gralog6.x Cluster.

can anyone give me the step by step guidances to set up gralog6.x Cluster ?

How did the opensearch embebed in graylog-datanode, how graylog-datanode controls opensearch ?

Is that i have to setup a real opensearch Cluster before setup a multinode graylog-datanode Cluster ?

Is that graylog-datanode need to be a Cluster or just the graylog-server need to be a Cluster ?

What are the roles in the Cluster architecture?

God bless you.

Hi @R1310328554,

I’d suggest updating the datanode package. You are using 6.1.3, but in 6.1.4 there is a fix that should solve your first problem with the datanode.

The error message Values less than -1 bytes are not supported suggests that you are running out of space on the disk where datanode wants to store its data. In datanodes before 6.1.4 opensearch tried to reserve 10GB for searchable snapshots cache. This could consume all of the available space, causing such problems and crashes.

The other stacktrace looks like something that can happen if you manually delete your mongodb data (while some nodes are still running?). I don’t think this step is necessary. Is there a reason why you have deleted datanode certificates and the mongodb content?

Best regards,
Tomas

thanks a lot!

i’ll try it later.

but still , there are lots of questions in my head. What is the relationship of opensearch and datanode , etc. as I asked later in above topic.

can you show some architecture diagrams of how a cluster works ?

thanks.

I manually deleted the datanode certificates and the mongodb content because I want to use the server’s web interface to configure the certificates of the second datanode, otherwise i cant access the server’s web interface which only shows in the first time login.

I have tried many many times and found this little trick useful when I want to reconfig a datanode.

I finally made it by started it over in a new enviroment.

the server error log is realy annoying and painful :

mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'Write operation error on server localhost:27017. Write error: WriteError{code=11000, message='E11000 duplicate key error collection: graylog.index_sets index: index_prefix_1 dup key: { index_prefix: "graylog" }', details={}}.'    

thanks all.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.