Cannot login to 2nd cluster node using LDAP authentication

1. Describe your incident:
Graylog is in the process of being deployed as a 2x nodes cluster and they share the same MongoDB instance running on node-1 for now, until I have the time to convert this single MongoDB instance into a replicaset.

I can connect with my LDAP/AD credentials on node-1 but on node-2 not always, as it uses to fail and this is shown on the login screen:

Error - the server returned: 503 - There was an error fetching a resource: . Additional information: Authentication service unavailable

The GL log shows this more extensive output:

2022-07-12T11:18:46.717+02:00 ERROR [LDAPAuthServiceBackend] LDAP error
com.unboundid.ldap.sdk.LDAPException: Simple bind operations are not allowed to contain a bind DN without a password.

[ 
 trimmed for clarity 
 ]

2022-07-12T11:18:46.722+02:00 INFO [SessionCreator] Session creation failed due to authentication service being unavailable. Actor: “urn:graylog:user:foobar@domain.tld”

2. Describe your environment:

  • OS Information: Ubuntu 20.04 LTS

  • Package Version: Graylog 4.3.2, OpenSearch 1.3.3

  • Service logs, configurations, and environment variables:

Graylog config file:

is_master = true

node_id_file = /etc/graylog/server/node-id

password_secret = XXXXXXXXXXXXXXXXXXX

root_password_sha2 = XXXXXXXXXXXXXXXXXXX

bin_dir = /usr/share/graylog-server/bin

data_dir = /var/lib/graylog-server

plugin_dir = /usr/share/graylog-server/plugin

http_bind_address = 0.0.0.0:9000

elasticsearch_version = 7

elasticsearch_hosts = http://admin:admin@node-1:9200,http://admin:admin@node-2:9200,http://admin:admin@node-3:9200

rotation_strategy = count

elasticsearch_max_docs_per_index = 20000000

elasticsearch_max_number_of_indices = 20

retention_strategy = delete

elasticsearch_shards = 4
elasticsearch_replicas = 0

elasticsearch_index_prefix = graylog

allow_leading_wildcard_searches = false

allow_highlighting = false

elasticsearch_analyzer = standard

output_batch_size = 500

output_flush_interval = 1

output_fault_count_threshold = 5
output_fault_penalty_seconds = 30

processbuffer_processors = 5
outputbuffer_processors = 3

processor_wait_strategy = blocking

ring_size = 65536

inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking

message_journal_enabled = true

message_journal_dir = /var/lib/graylog-server/journal

lb_recognition_period_seconds = 3

mongodb_uri = mongodb://node-1:27017/graylog

mongodb_max_connections = 1000

mongodb_threads_allowed_to_block_multiplier = 5

proxied_requests_thread_pool_size = 32

prometheus_exporter_enabled = true

prometheus_exporter_bind_address = 127.0.0.1:9090

node-2 connects to both OpenSearch and MongoDB:

2022-07-12T11:18:22.839+02:00 INFO  [MongoDBPreflightCheck] Connected to MongoDB version 5.0.9
2022-07-12T11:18:22.952+02:00 INFO  [SearchDbPreflightCheck] Connected to (Elastic/Open)Search version <OpenSearch:1.3.3>

3. What steps have you already taken to try and solve the problem?

The fix is to log in as the admin user on node-2 and re-enter the LDAP bind password
 which should be already set on node-1, stored on MongoDB and accessed from node-2


4. How can the community help?

I suppose that logging in from all nodes in a cluster should be possible and not only from the master.
If that is the case, could you please tell me if am I overseeing the obvious?

The final picture is to have both nodes behind an Azure LB but LDAP authentication must work on both nodes first.

Thanks a lot in advance!

Hello,

Correct me if I’m wrong, from your settings and statement above see if I can understand this setup.

Ok I get this, two Graylog servers with one MongoDb on node-01, kind of odd but I get it.

You have a three node OpenSearch cluster?

As for the Log error

This shows me that your Graylog server bind to the Domain Name server ( DNS) is not allowing a simple bind, you need a username and password.

Think I’m confused :laughing: You can login the Web UI but you cant login in Web UI from Node-02?

I assume you have a graylog node-01 as Master and Graylog Node-02 is not a Master? or there both masters?

My two node cluster Graylog/Mongodb configuration file looks like this.

NODE-01

http_bind_address = 192.168.1.92:9000
elasticsearch_hosts = http://192.168.1.95:9200, http://192.168.1.96:9200, http://192.168.1.97:9200
mongodb_uri = mongodb://192.168.1.92:27017,192.168.1.93:27017/graylog?replicaSet=replica01

NODE-02

http_bind_address = 192.168.1.93:9000
elasticsearch_hosts = http://192.168.1.95:9200, http://192.168.1.96:9200, http://192.168.1.97:9200
mongodb_uri = mongodb://192.168.1.93:27017,192.168.1.92:27017/graylog?replicaSet=replica01

I haven’t used OpenSearch yet but it shouldn’t make a difference on LDAP connection.

I do need to ask a question, Why do you need to login the second node anyways. One node should control the others, hence “master” and if a load balancer is in front it shouldn’t make a difference. The cluster in this case, I do believe is for redundancy?

That is correct, the error above has something to do with the AD/LDAP configuration. In my case I do not use a wild card for IP address bind. I use static IP addresses and insure each address has A, AAAA, and PTR type DNS records. Also each node is in my /etc/hosts file and configured. That’s just me though.

NODE-01 ( /etc/hosts)

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.95 es-node-01
192.168.1.96 es-node-02
192.168.1.97 es-node-03

192.168.1.92 GL/Mongo-01
192.168.1.93 GL/Mongo-02

NODE-02 ( /etc/hosts)

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.95 es-node-01
192.168.1.96 es-node-02
192.168.1.97 es-node-03

192.168.1.92 GL/Mongo-01
192.168.1.93 GL/Mongo-02

EDIT: I forgot to mention. In my domain I created a user account service for LDAP connection as shown below. The user service account /w password is able to make a connection to AD DC. But this is a lab I’m not using certs.

Example.

1 Like

Hello @gsmith !

thanks for your reply!

You have a three node OpenSearch cluster?

Yes. Graylog 4.3.x already supports it officially :slight_smile:

This shows me that your Graylog server bind to the Domain Name server ( DNS) is not allowing a simple bind, you need a username and password.

No, not DNS
 LDAP! :slight_smile:

http_bind_address = 192.168.1.92:9000

if I set the “http_bind_address” to the node’s IP, I get a “5xx error - bad gateway” but works for me with either 127.0.0.1:9000 or 0.0.0.0:9000

This is the nginx config:

# cat /etc/nginx/sites-enabled/default 
server {
        listen 80 default_server;
        server_name _;
        return 301 https://$host$request_uri;
}

server {
        listen 443 ssl http2;

        server_name new-graylog-server.tld;
        ssl_certificate /etc/nginx/ssl/graylog.crt;
        ssl_certificate_key /etc/nginx/ssl/private/graylog.key;

        ssl_session_timeout 1d;
        ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
        ssl_session_tickets off;

        # modern configuration
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
        ssl_prefer_server_ciphers off;

        # HSTS (ngx_http_headers_module is required) (63072000 seconds)
        add_header Strict-Transport-Security "max-age=63072000" always;

        # OCSP stapling
        ssl_stapling on;
        ssl_stapling_verify on;

        resolver 127.0.0.1;

        location / {
          proxy_set_header Host $http_host;
          proxy_set_header X-Forwarded-Host $host;
          proxy_set_header X-Forwarded-Server $host;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header X-Graylog-Server-URL https://$server_name/;
          proxy_pass       http://127.0.0.1:9000;
        }
}

/etc/hosts is populated on both nodes in a similar way to yours.

node-1 is set as “is_master = true” and node-2 is set as “is_master = false” - otherwise a nasty error message will show up in the GUI. :slight_smile:

I do need to ask a question, Why do you need to login the second node anyways. One node should control the others, hence “master” and if a load balancer is in front it shouldn’t make a difference. The cluster in this case, I do believe is for redundancy?

I’m happy accessing only 1 node. Some colleagues at work complained when they could access node-2. Indeed, the cluster setup is for redundancy.

I also use a dedicated service account to connect to LDAP.

As I said previously, it seems like the password for that LDAP service account is lost and I must re-enter it if I want it to work
 which somehow is the reason for my first post on this topic :slight_smile:

As for the MongoDB config:

mongodb_uri = mongodb://192.168.1.93:27017,192.168.1.92:27017/graylog?replicaSet=replica01

Did yo follow any special document specific to Graylog to create that replicaset, or did you use this one? https://www.mongodb.com/docs/manual/tutorial/convert-standalone-to-replica-set/

Thanks again!

Hello,

Oh nice so you have Nginx in front of Graylog.

My apologies, all the DMZ’s I work in the DNS LDAP/AD are normally on the same server. One checks IP address ( direct traffic) and the other checks Credentials for the domain.

Understood, This can be fix thou, its a configuration issue.
Here is my logical diagram, and the only time I use Localhost/127.0.0.1 is when all the software is on one machine. Or I’m deal with Docker.

If I understand this correct. The LDAP credentials that was configured originally get lost/removed from Graylog? If this is correct I would check MongoDb it holds all metadata (i.e. users, credentials, etc
)

kind of, I did follow MongoDb documentation starting from here

Here is my personal Documentation on this subject. There are Three parts, One the MongoDb configuration file, second internal Database configuration and last adding them in Graylog configuration file This was for MongoDb-4.0.

This is one of two MongoDb configuration file.
NOTE: keep an eye out on line indents, naming convention’s.

# mongod.conf
# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/
# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log
# Where and how to store data.
storage:
  dbPath: /var/lib/mongo
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:
# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile
  #timeZoneInfo: /usr/share/zoneinfo

#Network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1  # Listen to local interface only, comment to listen on all interfaces.
  # bindIp: 0.0.0.0

security:
    authorization: enabled
operationProfiling:
 slowOpThresholdMs: 5
 mode: all

replication:
 replSetName: "replica01"
#sharding:

[MongoDB Replica Set initiate]
We will use the lab-graylog-001.domain.com server as ‘PRIMARY’ node, and ’ lab-graylog-002.domain.com ’ and ’ lab-graylog-003.domain.com ’ as ‘SECONDARY’ nodes.

Execute command

shell# mongo
shell# rs.initiate()

Add the ’ lab-graylog-002.domain.com ’ and ’ lab-graylog-003.domain.com ’ nodes to the replica sets.

Execute command

shell#rs.add("lab-graylog-002.domain.com ")
Shell#rs.add("lab-graylog-003.domain.com ")

Check the replica sets status with the rs query below.

Shell# rs.status()

Query to check the status on ‘lab-graylog-001.domain.com :

Shell# rs.isMaster()  // Should Show “isMaster” =True

NOTE: Enable reading from the ‘SECONDARY’ node with the query ‘rs.slaveOk()’

You can find more here

Maybe some one else here has a better idea.

Hope that helps

Hello,

I have exactly the same issue on my Graylog cluster (2 nodes):
My environment is:

- OS Information: Debian 11.3
- Package Version: Graylog 4.3.2, Elasticsearch 7.10.2, MongoDB 4.2.20
- I have a replica set for MongoDB with 3 nodes (I had 2 nodes initially, I added a node to see if the problem would be solved but not at all)

When I configure AD authentication on Graylog web interface, I can access on node-1 with my AD credentials but never on node-2. I have the follow error message on node-2:

Error - the server returned: 503 - There was an error fetching a resource: Service Unavailable. Additional information: Authentication service unavailable

And on Graylog server logs:

2022-07-15T10:14:52.878+02:00 ERROR [ADAuthServiceBackend] ActiveDirectory error
com.unboundid.ldap.sdk.LDAPBindException: 80090308: LdapErr: DSID-0C090439, comment: AcceptSecurityContext error, data 52e, v4563
        at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:2381) ~[graylog.jar:?]
        at org.graylog.security.authservice.ldap.UnboundLDAPConnector.connect(UnboundLDAPConnector.java:141) ~[graylog.jar:?]
        at org.graylog.security.authservice.backend.ADAuthServiceBackend.authenticateAndProvision(ADAuthServiceBackend.java:89) ~[graylog.jar:?]
        at org.graylog.security.authservice.AuthServiceAuthenticator.authenticate(AuthServiceAuthenticator.java:102) ~[graylog.jar:?]
        at org.graylog.security.authservice.AuthServiceAuthenticator.authenticate(AuthServiceAuthenticator.java:65) ~[graylog.jar:?]
        at org.graylog2.security.realm.UsernamePasswordRealm.doGetAuthenticationInfo(UsernamePasswordRealm.java:93) ~[graylog.jar:?]
        at org.graylog2.security.realm.UsernamePasswordRealm.doGetAuthenticationInfo(UsernamePasswordRealm.java:71) ~[graylog.jar:?]
        at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:571) ~[graylog.jar:?]
        at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doMultiRealmAuthentication(ModularRealmAuthenticator.java:225) ~[graylog.jar:?]
        at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:275) ~[graylog.jar:?]
        at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198) ~[graylog.jar:?]
        at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106) ~[graylog.jar:?]
        at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:275) ~[graylog.jar:?]
        at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:260) ~[graylog.jar:?]
        at org.graylog2.shared.security.SessionCreator.login(SessionCreator.java:87) ~[graylog.jar:?]
        at org.graylog2.rest.resources.system.SessionsResource.newSession(SessionsResource.java:140) ~[graylog.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
2022-07-15T10:14:52.879+02:00 INFO  [SessionCreator] Session creation failed due to authentication service being unavailable. Actor: "urn:graylog:user:xxxxxxxx"

When I try to configure AD auth on node-2, I can access it but node-1 auth no longer works in turn. So I think that AD configurations is accessible for only one node in cluster.

When I check Graylog stored configuration on all MongoDB nodes AD authentication configuration is the same for all nodes under the replica set

I would like to configure AD authentication on both Graylog nodes.

Thank you for your help!

@paul.girodroux

 ERROR [ADAuthServiceBackend] ActiveDirectory error com.unboundid.ldap.sdk.LDAPBindException

There is a post here about Windows Server AD DC.

Node-01 is_master = true ?
Node-02 is_master = false?

You could do a couple test.
1.Login with the configured admin/root users for graylog.
If this works it might have something to do with settings/configuration.
2.If able to execute this try switching the master to node-02.
If that works then we know its the settings.

MongoDb replica set one write and secondary’s reads. So if both of the MongoDb database have the same credentials for AD then It has something to do with Graylog Config file settings.

EDIT: I don’t think I ask this question but on the Web UI , under System/Node do you see both Graylog servers? Just curious
 The pic below shows the Master with Gold star.

Hello @paul.girodroux

my 2x nodes show the very same symptoms you describe :frowning:
When I configure node-2 (= I only need to re-add the LDAP password) and I can access it, node-1 doesn’t work :frowning:

I can live with accessing only node-1 but I’m afraid that this will not work once I put both nodes behind a load-balancer
 I stopped the Graylog service on node-1 hoping that node-2 would take over the LDAP authentication but it did not
 :-/

Ideas on how to solve this?

Thanks!

1 Like

Hello @gsmith

My configuration is Node-1 master and Node-2 not master.
I tried to switch this parameter but it didn’t work.

When I tried to connect on Node-1, AD credentials work but on Node-2 I have the same issue as before.

I have new logs this time on Node-2 :

javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        at com.sun.crypto.provider.CipherCore.unpad(CipherCore.java:975) ~[?:?]
        at com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1056) ~[?:?]
        at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:853) ~[?:?]
        at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446) ~[?:?]
        at javax.crypto.Cipher.doFinal(Cipher.java:2202) ~[?:?]
        at org.graylog2.security.AESTools.decrypt(AESTools.java:88) ~[graylog.jar:?]
        at org.graylog2.security.encryption.EncryptedValueService.decrypt(EncryptedValueService.java:57) ~[graylog.jar:?]
        at org.graylog.security.authservice.ldap.UnboundLDAPConnector.connect(UnboundLDAPConnector.java:139) ~[graylog.jar:?]
        at org.graylog.security.authservice.backend.ADAuthServiceBackend.authenticateAndProvision(ADAuthServiceBackend.java:89) ~[graylog.jar:?]
        at org.graylog.security.authservice.AuthServiceAuthenticator.authenticate(AuthServiceAuthenticator.java:102) ~[graylog.jar:?]
        at org.graylog.security.authservice.AuthServiceAuthenticator.authenticate(AuthServiceAuthenticator.java:65) ~[graylog.jar:?]
        at org.graylog2.security.realm.UsernamePasswordRealm.doGetAuthenticationInfo(UsernamePasswordRealm.java:93) ~[graylog.jar:?]
        at org.graylog2.security.realm.UsernamePasswordRealm.doGetAuthenticationInfo(UsernamePasswordRealm.java:71) ~[graylog.jar:?]
        at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:571) ~[graylog.jar:?]
        at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doMultiRealmAuthentication(ModularRealmAuthenticator.java:225) ~[graylog.jar:?]
        at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:275) ~[graylog.jar:?]
        at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198) ~[graylog.jar:?]
        at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106) ~[graylog.jar:?]
        at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:275) ~[graylog.jar:?]
        at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:260) ~[graylog.jar:?]
        at org.graylog2.shared.security.SessionCreator.login(SessionCreator.java:87) ~[graylog.jar:?]
        at org.graylog2.rest.resources.system.SessionsResource.newSession(SessionsResource.java:140) ~[graylog.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
2022-07-18T09:43:02.783+02:00 ERROR [ADAuthServiceBackend] ActiveDirectory error
com.unboundid.ldap.sdk.LDAPException: Simple bind operations are not allowed to contain a bind DN without a password.
        at com.unboundid.ldap.sdk.SimpleBindRequest.process(SimpleBindRequest.java:555) ~[graylog.jar:?]
        at com.unboundid.ldap.sdk.LDAPConnection.processBindOperation(LDAPConnection.java:4540) ~[graylog.jar:?]
        at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:2373) ~[graylog.jar:?]
        at org.graylog.security.authservice.ldap.UnboundLDAPConnector.connect(UnboundLDAPConnector.java:141) ~[graylog.jar:?]
        at org.graylog.security.authservice.backend.ADAuthServiceBackend.authenticateAndProvision(ADAuthServiceBackend.java:89) ~[graylog.jar:?]
        at org.graylog.security.authservice.AuthServiceAuthenticator.authenticate(AuthServiceAuthenticator.java:102) ~[graylog.jar:?]
        at org.graylog.security.authservice.AuthServiceAuthenticator.authenticate(AuthServiceAuthenticator.java:65) ~[graylog.jar:?]
        at org.graylog2.security.realm.UsernamePasswordRealm.doGetAuthenticationInfo(UsernamePasswordRealm.java:93) ~[graylog.jar:?]
        at org.graylog2.security.realm.UsernamePasswordRealm.doGetAuthenticationInfo(UsernamePasswordRealm.java:71) ~[graylog.jar:?]
        at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:571) ~[graylog.jar:?]
        at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doMultiRealmAuthentication(ModularRealmAuthenticator.java:225) ~[graylog.jar:?]
        at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:275) ~[graylog.jar:?]
        at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198) ~[graylog.jar:?]
        at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106) ~[graylog.jar:?]
        at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:275) ~[graylog.jar:?]
        at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:260) ~[graylog.jar:?]
        at org.graylog2.shared.security.SessionCreator.login(SessionCreator.java:87) ~[graylog.jar:?]
        at org.graylog2.rest.resources.system.SessionsResource.newSession(SessionsResource.java:140) ~[graylog.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) [graylog.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) [graylog.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [graylog.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) [graylog.jar:?]
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?]
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]
2022-07-18T09:43:02.784+02:00 INFO  [SessionCreator] Session creation failed due to authentication service being unavailable. Actor: "urn:graylog:user:xxxxxxxxx"

I tried to reconfigure AD auth on both servers but same issue as before even after I switched master configuration.
I think the issue comes from the password which is not stored in both MongoDB nodes but when I checked on both MongoDB database inside the replica set, I saw exactly the same AD configuration.

Thanks!

EDIT : here it is the page with nodes

In our case, we only use 1 MongoDB instance (it will be converted to a replicaset eventually) so theory says that the LDAP credentials are already set


I stopped Graylog on node-1, configured node-2 as master and restarted Graylog there
 still, LDAP does not work on node-2 :frowning:

My guess is that the LDAP credentials are encrypted using node-1’s salt or similar, so when you set the LDAP password on node-2, you cannot access from node-1


> db.auth_service_backends.find().pretty()
{
        "_id" : ObjectId("XXXXXXXXXXXXXXXXXX"),
        "title" : "LDAP",
        "description" : "",
        "default_roles" : [
                "XXXXXXXXXXXXXXXXXXX"
        ],
        "config" : {
                "type" : "ldap",
                "servers" : [
                        {
                                "host" : "our-ldap-server",
                                "port" : 636
                        }
                ],
                "transport_security" : "tls",
                "verify_certificates" : false,
                "system_user_dn" : "service-account@domain.tld",
                "system_user_password" : {
                        "encrypted_value" : "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
                        "salt" : "YYYYYYYYYYYYYY"
                },
                "user_search_base" : "XXXXXXXXXXXXXXXXXXXXXX",
                "user_search_pattern" : "XXXXXXXXXXXXXXXXXXXXXXXX",
                "user_unique_id_attribute" : "userprincipalname",
                "user_name_attribute" : "userprincipalname",
                "user_full_name_attribute" : "cn"
        }
}
>

If that’s the case, I don’t know how to solve it


We don’t allow password-less LDAP binding


Ideas?

Thanks!

[SOLVED] I found the solution to the problem and it was right before my eyes :slight_smile:

# You MUST set a secret to secure/pepper the stored user passwords here. Use at least 64 characters.
# Generate one by using for example: pwgen -N 1 -s 96
# ATTENTION: This value must be the same on all Graylog nodes in the cluster.
# Changing this value after installation will render all user sessions and encrypted values in the database invalid. (e.g. encrypted access token
s)
password_secret = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Copy the password_secret from node-1 to node-2, restart Graylog on node-2 and you’ll be all set.

HTH

2 Likes

It works for me too !

Thanks a lot for the solution ! :slight_smile:

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.