I’m trying out Graylog for the first time. I’ve managed to install Elasticsearch, MongoDB, Graylog server and the Datanode on the same machine. I read somewhere that it’s not ideal but this is for a relatively small deployment.
Once the installation was all done, and I logged in to the Web UI and from there I created the CA bundle and then issued a certificate for the datanode that is on the same machine. But nothing happens and when I looked at the logs, I see the following:
Server log:
2025-01-07T15:24:09.517+02:00 ERROR [CertificateExchangeImpl] Failed to sign CSR for node, skipping it for now.
java.lang.RuntimeException: java.lang.NullPointerException: Cannot invoke “org.bouncycastle.pkcs.PKCS10CertificationRequest.getSubject()” because the return value of “org.graylog2.cluster.certificates.CertificateSigningRequest.request()” is null
at org.graylog.security.certutil.CaKeystore.signCertificateRequest(CaKeystore.java:75) ~[graylog.jar:?]
at org.graylog2.bootstrap.preflight.GraylogCertificateProvisionerImpl.lambda$runProvisioning$0(GraylogCertificateProvisionerImpl.java:61) ~[graylog.jar:?]
at org.graylog2.cluster.certificates.CertificateExchangeImpl.signPendingCertificateRequests(CertificateExchangeImpl.java:102) [graylog.jar:?]
at org.graylog2.bootstrap.preflight.GraylogCertificateProvisionerImpl.runProvisioning(GraylogCertificateProvisionerImpl.java:61) [graylog.jar:?]
at org.graylog2.bootstrap.preflight.GraylogCertificateProvisioningPeriodical.doRun(GraylogCertificateProvisioningPeriodical.java:40) [graylog.jar:?]
at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99) [graylog.jar:?]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.NullPointerException: Cannot invoke “org.bouncycastle.pkcs.PKCS10CertificationRequest.getSubject()” because the return value of “org.graylog2.cluster.certificates.CertificateSigningRequest.request()” is null
at org.graylog.security.certutil.CaKeystore.signCertificateRequest(CaKeystore.java:67) ~[graylog.jar:?]
Datanode log:
2025-01-07T15:28:42.291+02:00 INFO [CustomCAX509TrustManager] CA changed, refreshing trust manager
2025-01-07T15:28:42.296+02:00 ERROR [graylog-eventbus] Exception thrown by subscriber method handleCertificateAuthorityChange(org.graylog.security.certutil.CertificateAuthorityChangedEvent) on subscriber org.graylog2.security.CustomCAX509TrustManager@2a99ca99 when dispatching event: CertificateAuthorityChangedEvent
java.lang.IllegalArgumentException: Illegal base64 character 22
at java.base/java.util.Base64$Decoder.decode0(Unknown Source) ~[?:?]
at java.base/java.util.Base64$Decoder.decode(Unknown Source) ~[?:?]
at java.base/java.util.Base64$Decoder.decode(Unknown Source) ~[?:?]
at java.base/java.util.Optional.map(Unknown Source) ~[?:?]
at org.graylog.security.certutil.CaPersistenceService.readFromDatabase(CaPersistenceService.java:205) ~[graylog2-server-6.1.4.jar:?]
at org.graylog.security.certutil.CaPersistenceService.loadKeyStore(CaPersistenceService.java:187) ~[graylog2-server-6.1.4.jar:?]
at org.graylog.security.certutil.CaTruststoreImpl.getTrustStore(CaTruststoreImpl.java:55) ~[graylog2-server-6.1.4.jar:?]
at org.graylog2.security.CustomCAX509TrustManager.refresh(CustomCAX509TrustManager.java:58) ~[graylog2-server-6.1.4.jar:?]
at org.graylog2.security.CustomCAX509TrustManager.handleCertificateAuthorityChange(CustomCAX509TrustManager.java:51) ~[graylog2-server-6.1.4.jar:?]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:85) ~[guava-33.3.1-jre.jar:?]
at com.google.common.eventbus.Subscriber$SynchronizedSubscriber.invokeSubscriberMethod(Subscriber.java:142) ~[guava-33.3.1-jre.jar:?]
at com.google.common.eventbus.Subscriber.lambda$dispatchEvent$0(Subscriber.java:71) ~[guava-33.3.1-jre.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:259) [metrics-core-4.2.28.jar:4.2.28]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2025-01-07T15:28:51.301+02:00 INFO [CsrRequesterImpl] Triggered certificate signing request for this datanode
Here are the configuration changes on the machine:
Hi @rubin.j
My first idea is to check if your graylog server and datanode share the same password_secret. This is used on both sides to encrypt/decrypt CA and certificate signing requests and must be identical in both configuration files.
Your logs suggest that your datanode has troubles to read the CA keystore and the graylog server can’t read certificate signing requests. Both sides rely on the password secret to decrypt these values from the DB.
So far I don’t think this problem could be caused by running both on the same machine, even if it’s not really production-ready setup and we don’t recommend it.
Thanks for your reply. I tried using the same password_secret but that led to a different problem. Seeing as everyone is not recommending the server and datanode be on the same machine, I started over.
Now there are 2 machines, and I went ahead with the installation and configuration (with a little help from ChatGPT). As soon as I got the configurations correct, the server web UI changed completely. It’s no longer showing the initial setup page and instead showing the personal page. There is also an error mentioning that there’s no input coming from the datanode:
Not sure what to do now. I get the feeling that this might have something to do with Elasticserach. The elasticsearch configs for the server and datanode are below:
I know this is no longer the original issue, and if you want me to open a new case, I’ll do that. But I’d really appreciate it if we could continue on this case here. Thank you so much for your patience and support!
You don’t need elasticsearch at all, the datanode implements all the indexing and management logic instead (under the hood, datanode is a managed opensearch, a fork of the elasticsearch).
So I’d continue with removal of the elasticsearch. Then you can remove the elasticsearch_hosts property in the server configuration. Graylog server will autodetect and use your datanode instead, just restart the server after you change the configuration.
Otherwise the screenshot looks good. It just tells you that you haven’t configured (in the UI) any inputs yet, so there is no data being ingested in your instance. This is the default when you start with a fresh installation. You can configure your inputs under system - inputs.
Disabling the elasticsearch configuration in the server seems to have caused it to crash completely. So should I start over completely and not bother with the installation of Elasticsearch? Also, do both the server and datanode require MongoDB?
My ultimate goal here is to use Graylog as a log monitoring and alerting system for several cryptocoin nodes. The idea is to monitor the logs of the cryptocoin services (like bitcoin) so that the corresponding dashboard to a coin node will highlight any errors in the logs and send email alerts as well. Is this feasible with Graylog? I suppose I should have asked this in the first place!
You use case sounds like a very good match to the features that Graylog offers.
I’d recommend you start again. First remove the Elasticsearch, as you don’t need it. The mongodb is shared between both data node and graylog server. It’s the central sync point, enables autodiscovery of components, shares CA configuration and many more things. This also means your mongodb connection string has to be the same in both data node and graylog server. If you won’t reinstall it, at least remove all data in it, so graylog can start without any previous configuration.
If you have mongodb running, you can install data node and graylog server. Configure the mongodb connection, password secret and all other required fields.
When the graylog server starts, it will give you access to the preflight interface, where the data node will be autodetected and secured by a CA, either self-signed or your own, depending on your needs. After that, you should get a running system. There you can configure inputs, collect data and configure your dashboards and alerts.
Thanks for your reply, Tomas. Alright, I’ll start over then. But before that, please confirm whether MongoDb should be installed on the server or on the datanode?
I the perfect world, you’d have a cluster of 3+ mongodb nodes on different machines If you are playing with the setup, I don’t think it matters where you’ll have your single mongodb instance running. There are no significant performance requirements for the database, so you can run it where it’s convenient for you, as long as it stays reachable for both data node and graylog server.
You can also have all three components, mongodb, data node and graylog server on one machine. Technically, it will work. It’s not recommended, but you can do it. Some people are using docker and 3 containers in one machine, one container for each service.
If you’ll be happy with your setup, you can always migrate the data later, splitting those services to their own machines.
So I redid the Grayog server from scratch. I left the Datanode as is but removed Elasticsearch. I’ve got the server up and running and I could login with the prelim password in the log. But it doesn’t look like the server is detecting the datanode:
Here are the relevant settings on the server:
http_bind_address = 0.0.0.0:9000
http_publish_uri = http://act-monlog-01.act.local:9000/
mongodb_uri = mongodb://localhost/graylog
And here are the settings on the node. I’ve added in extra stuff hoping to get the server-node communication going, but no dice.
bind_address = 0.0.0.0
hostname = act-monlog-02.act.local
datanode_http_port = 8999
opensearch_http_port = 9200
opensearch_transport_port = 9300
node_name = act-monlog-02
I’ve ensured that the password_secret is the same on both the server and node. Any idea why the node isn’t being picked up?
If you are playing with the setup, I don’t think it matters where you’ll have your single mongodb instance running. There are no significant performance requirements for the database, so you can run it where it’s convenient for you, as long as it stays reachable for both data node and graylog server.