LDAP auth broken with recent patch? FIPS mode breaks openjdk crypto

1. Describe your incident:

Seemingly after a recent update (maybe log4J) LDAP has failed. Test Server Configuration fails. This has worked for many months with no changes made.

2. Describe your environment:

  • OS Information: RHEL 8.x

  • Package Version: Graylog 4.2.3+553fadb

  • Service logs, configurations, and environment variables:

    There was an error fetching a resource: Bad Request. Additional information: Null value (through reference chain: org.graylog.security.authservice.test.AutoValue_AuthServiceBackendTestRequest$Builder[“backend_configuration”]->org.graylog.security.authservice.$AutoValue_AuthServiceBackendDTO$Builder[“config”]->org.graylog.security.authservice.backend.AutoValue_ADAuthServiceBackendConfig$Builder[“system_user_password”])
    false

3. What steps have you already taken to try and solve the problem?

4. How can the community help?

Hello

Is this installation Open or Enterprise version? Can I ask what you have tried to do to resolve this issue?

We have a free Enterprise license. So far I have tried nothing. This has worked for months and started failing after the last Graylog update yesterday. Using the LDAP authentication wizard fails the Test Server Configuration showing the included message above. I didn’t notice because my cookie had not expired until late today. I see a server.conf.rpmnew, but there is nothing there aside from the name/pass/server fields that require editing. The rest of the diffs are commented info.

Have you checked Graylog log file to see if anything that would pertain to this issue?

EDIT: The more information we have the better we can troubleshoot the issue

I did a quick update in my Lab from version 4.2.2 to 4.2.3 using my Active directory user for Graylog.
I’m not sure how you set up your LDAP but here is mine.

Unfortunately I cant replicate that issue.

I’ll be back on it in the morning, but one difference is on mine Verify Certificate is checked. Although neither TLS or StartTLS are. Also pam/sssd also use LDAP and they still work. Totally different mechanisms of course.

I understand , no problem.

When you get back on maybe this post below will help troubleshoot your issue. To sum it up it seams he had the same error as you do.

Too bad there was never an answer.

Hello

Agree, must people are like that. Either they get what they need and just drop off.

Did another quick test in my lab. This is without a ladp_user and password.

Not really sure what’s is causing your issue.

Your LDAP must allow anonymous lookup.

Yes, in my lab our environment. Might want to check your LADP log files. Perhaps you can see Graylog trying to connect.

The original error you posted suggests a null field associated with system_user_password. For the User DN, you can try using the full username such as Auth-account@mylocaldomain.tld

Also double check that your enterprise plugins are updated - you can use this command to check (I think)

yum list installed | grep -E ".*(elasticsearch|graylog|mongo).*"

Full user name made no difference.

[root@graylog server]# dnf list installed | grep -E “.(elasticsearch|graylog|mongo).
elasticsearch-oss.x86_64 7.10.2-1 @elasticsearch-7.x
graylog-4.2-repository.noarch 1-4 @graylog
graylog-enterprise-integrations-plugins.noarch 4.2.3-1 @graylog
graylog-enterprise-plugins.noarch 4.2.3-1 @graylog
graylog-integrations-plugins.noarch 4.2.3-1 @graylog
graylog-server.noarch 4.2.3-1 @graylog
graylog-sidecar-repository.noarch 1-2 @System
mongodb-org.x86_64 4.2.17-1.el8 @mongodb-org-4.2
mongodb-org-mongos.x86_64 4.2.17-1.el8 @mongodb-org-4.2
mongodb-org-server.x86_64 4.2.17-1.el8 @mongodb-org-4.2
mongodb-org-shell.x86_64 4.2.17-1.el8 @mongodb-org-4.2
mongodb-org-tools.x86_64 4.2.17-1.el8 @mongodb-org-4.2

Rather than full user name, I should have put userPrincipalName

Test to see if your RHEL machine can see the LDAP server - maybe use ldapwhoami … Serverfault has a good example test command

I don’t use redhat, should I be suggesting dnf… rather than yum… ?

The pam/sssd login uses LDAP, so LDAP works for pam.

Here is the full log for a failed login:

2021-12-16T13:18:24.508-05:00 ERROR [AESTools] Could not encrypt value.
java.security.NoSuchProviderException: No such provider: SunJCE
	at javax.crypto.Cipher.getInstance(Cipher.java:596) ~[?:1.8.0_312]
	at org.graylog2.security.AESTools.encrypt(AESTools.java:57) ~[graylog.jar:?]
	at org.graylog2.security.encryption.EncryptedValueService.encrypt(EncryptedValueService.java:45) ~[graylog.jar:?]
	at org.graylog2.security.realm.UsernamePasswordRealm.doGetAuthenticationInfo(UsernamePasswordRealm.java:92) ~[graylog.jar:?]
	at org.graylog2.security.realm.UsernamePasswordRealm.doGetAuthenticationInfo(UsernamePasswordRealm.java:71) ~[graylog.jar:?]
	at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:571) ~[graylog.jar:?]
	at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doMultiRealmAuthentication(ModularRealmAuthenticator.java:225) ~[graylog.jar:?]
	at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:275) ~[graylog.jar:?]
	at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198) ~[graylog.jar:?]
	at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106) ~[graylog.jar:?]
	at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:275) ~[graylog.jar:?]
	at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:260) ~[graylog.jar:?]
	at org.graylog2.shared.security.SessionCreator.create(SessionCreator.java:82) ~[graylog.jar:?]
	at org.graylog2.rest.resources.system.SessionsResource.newSession(SessionsResource.java:142) ~[graylog.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_312]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_312]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_312]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_312]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) [graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) [graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) [graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) [graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) [graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [graylog.jar:?]

	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) [graylog.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?]

	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2021-12-16T13:18:24.509-05:00 ERROR [UsernamePasswordRealm] Unhandled authentication error
java.lang.NullPointerException: Null value
	at org.graylog2.security.encryption.AutoValue_EncryptedValue$Builder.value(AutoValue_EncryptedValue.java:96) ~[graylog.jar:?]
	at org.graylog2.security.encryption.EncryptedValueService.encrypt(EncryptedValueService.java:45) ~[graylog.jar:?]
	at org.graylog2.security.realm.UsernamePasswordRealm.doGetAuthenticationInfo(UsernamePasswordRealm.java:92) ~[graylog.jar:?]
	at org.graylog2.security.realm.UsernamePasswordRealm.doGetAuthenticationInfo(UsernamePasswordRealm.java:71) ~[graylog.jar:?]
	at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:571) ~[graylog.jar:?]
	at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doMultiRealmAuthentication(ModularRealmAuthenticator.java:225) ~[graylog.jar:?]
	at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:275) ~[graylog.jar:?]
	at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198) ~[graylog.jar:?]
	at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106) ~[graylog.jar:?]
	at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:275) ~[graylog.jar:?]
	at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:260) ~[graylog.jar:?]
	at org.graylog2.shared.security.SessionCreator.create(SessionCreator.java:82) ~[graylog.jar:?]
	at org.graylog2.rest.resources.system.SessionsResource.newSession(SessionsResource.java:142) ~[graylog.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_312]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_312]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_312]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_312]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) [graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) [graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) [graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) [graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) [graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [graylog.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) [graylog.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2021-12-16T13:18:24.510-05:00 INFO  [SessionCreator] Invalid credentials in session create request. Actor: "urn:graylog:user:joe_user"

Hmmm I have seen a few things on how password_secret in server.conf is used for salting passwords… is that something that might have changed in your test configuration or maybe wonked in translation?

No, the hash has not changed. Non-LDAP logins work just fine.

Hello

I found this. Different version looks like same issue.

I don’t have illegal key size, so not the same issue. Seems like the root cause is No such provider: SunJCE. I did more testing with a different AD account, same issue. Tried with TLS, same issue.

OK, so I’m a nitwit. We had an audit last week and enabled FIPS mode on all Linux systems. I’m REALLY confident this is what killed LDAP login. Probably SunJCE can’t function in that cryptographic mode. I have not verified, but I’m pretty sure that is the cause. Of course there is no reference to FIPS in Graylog documentation.