LDAP auth broken with recent patch? FIPS mode breaks openjdk crypto

mntbighker · December 15, 2021, 11:25pm

1. Describe your incident:

Seemingly after a recent update (maybe log4J) LDAP has failed. Test Server Configuration fails. This has worked for many months with no changes made.

2. Describe your environment:

OS Information: RHEL 8.x
Package Version: Graylog 4.2.3+553fadb
Service logs, configurations, and environment variables:

There was an error fetching a resource: Bad Request. Additional information: Null value (through reference chain: org.graylog.security.authservice.test.AutoValue_AuthServiceBackendTestRequest$Builder[“backend_configuration”]->org.graylog.security.authservice.$AutoValue_AuthServiceBackendDTO$Builder[“config”]->org.graylog.security.authservice.backend.AutoValue_ADAuthServiceBackendConfig$Builder[“system_user_password”])
false

3. What steps have you already taken to try and solve the problem?

4. How can the community help?

gsmith · December 16, 2021, 1:24am

Hello

Is this installation Open or Enterprise version? Can I ask what you have tried to do to resolve this issue?

mntbighker · December 16, 2021, 3:06am

We have a free Enterprise license. So far I have tried nothing. This has worked for months and started failing after the last Graylog update yesterday. Using the LDAP authentication wizard fails the Test Server Configuration showing the included message above. I didn’t notice because my cookie had not expired until late today. I see a server.conf.rpmnew, but there is nothing there aside from the name/pass/server fields that require editing. The rest of the diffs are commented info.

gsmith · December 16, 2021, 3:08am

Have you checked Graylog log file to see if anything that would pertain to this issue?

EDIT: The more information we have the better we can troubleshoot the issue

gsmith · December 16, 2021, 3:29am

I did a quick update in my Lab from version 4.2.2 to 4.2.3 using my Active directory user for Graylog.
I’m not sure how you set up your LDAP but here is mine.

Unfortunately I cant replicate that issue.

mntbighker · December 16, 2021, 3:57am

I’ll be back on it in the morning, but one difference is on mine Verify Certificate is checked. Although neither TLS or StartTLS are. Also pam/sssd also use LDAP and they still work. Totally different mechanisms of course.

gsmith · December 16, 2021, 4:04am

I understand , no problem.

When you get back on maybe this post below will help troubleshoot your issue. To sum it up it seams he had the same error as you do.

mntbighker · December 16, 2021, 4:24am

Too bad there was never an answer.

gsmith · December 16, 2021, 4:36am

Hello

Agree, must people are like that. Either they get what they need and just drop off.

Did another quick test in my lab. This is without a ladp_user and password.

Not really sure what’s is causing your issue.

mntbighker · December 16, 2021, 4:49am

Your LDAP must allow anonymous lookup.

gsmith · December 16, 2021, 4:57am

Yes, in my lab our environment. Might want to check your LADP log files. Perhaps you can see Graylog trying to connect.

tmacgbay · December 16, 2021, 5:12pm

The original error you posted suggests a null field associated with system_user_password. For the User DN, you can try using the full username such as Auth-account@mylocaldomain.tld

Also double check that your enterprise plugins are updated - you can use this command to check (I think)

yum list installed | grep -E ".*(elasticsearch|graylog|mongo).*"

mntbighker · December 16, 2021, 5:43pm

Full user name made no difference.

[root@graylog server]# dnf list installed | grep -E “.(elasticsearch|graylog|mongo).”
elasticsearch-oss.x86_64 7.10.2-1 @elasticsearch-7.x
graylog-4.2-repository.noarch 1-4 @graylog
graylog-enterprise-integrations-plugins.noarch 4.2.3-1 @graylog
graylog-enterprise-plugins.noarch 4.2.3-1 @graylog
graylog-integrations-plugins.noarch 4.2.3-1 @graylog
graylog-server.noarch 4.2.3-1 @graylog
graylog-sidecar-repository.noarch 1-2 @System
mongodb-org.x86_64 4.2.17-1.el8 @mongodb-org-4.2
mongodb-org-mongos.x86_64 4.2.17-1.el8 @mongodb-org-4.2
mongodb-org-server.x86_64 4.2.17-1.el8 @mongodb-org-4.2
mongodb-org-shell.x86_64 4.2.17-1.el8 @mongodb-org-4.2
mongodb-org-tools.x86_64 4.2.17-1.el8 @mongodb-org-4.2

tmacgbay · December 16, 2021, 5:57pm

Rather than full user name, I should have put userPrincipalName

Test to see if your RHEL machine can see the LDAP server - maybe use ldapwhoami … Serverfault has a good example test command

I don’t use redhat, should I be suggesting dnf… rather than yum… ?

mntbighker · December 16, 2021, 6:26pm

The pam/sssd login uses LDAP, so LDAP works for pam.

Here is the full log for a failed login:

2021-12-16T13:18:24.508-05:00 ERROR [AESTools] Could not encrypt value.
java.security.NoSuchProviderException: No such provider: SunJCE
	at javax.crypto.Cipher.getInstance(Cipher.java:596) ~[?:1.8.0_312]
	at org.graylog2.security.AESTools.encrypt(AESTools.java:57) ~[graylog.jar:?]
	at org.graylog2.security.encryption.EncryptedValueService.encrypt(EncryptedValueService.java:45) ~[graylog.jar:?]
	at org.graylog2.security.realm.UsernamePasswordRealm.doGetAuthenticationInfo(UsernamePasswordRealm.java:92) ~[graylog.jar:?]
	at org.graylog2.security.realm.UsernamePasswordRealm.doGetAuthenticationInfo(UsernamePasswordRealm.java:71) ~[graylog.jar:?]
	at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:571) ~[graylog.jar:?]
	at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doMultiRealmAuthentication(ModularRealmAuthenticator.java:225) ~[graylog.jar:?]
	at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:275) ~[graylog.jar:?]
	at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198) ~[graylog.jar:?]
	at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106) ~[graylog.jar:?]
	at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:275) ~[graylog.jar:?]
	at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:260) ~[graylog.jar:?]
	at org.graylog2.shared.security.SessionCreator.create(SessionCreator.java:82) ~[graylog.jar:?]
	at org.graylog2.rest.resources.system.SessionsResource.newSession(SessionsResource.java:142) ~[graylog.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_312]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_312]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_312]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_312]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) [graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) [graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) [graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) [graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) [graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [graylog.jar:?]

	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) [graylog.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?]

	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2021-12-16T13:18:24.509-05:00 ERROR [UsernamePasswordRealm] Unhandled authentication error
java.lang.NullPointerException: Null value
	at org.graylog2.security.encryption.AutoValue_EncryptedValue$Builder.value(AutoValue_EncryptedValue.java:96) ~[graylog.jar:?]
	at org.graylog2.security.encryption.EncryptedValueService.encrypt(EncryptedValueService.java:45) ~[graylog.jar:?]
	at org.graylog2.security.realm.UsernamePasswordRealm.doGetAuthenticationInfo(UsernamePasswordRealm.java:92) ~[graylog.jar:?]
	at org.graylog2.security.realm.UsernamePasswordRealm.doGetAuthenticationInfo(UsernamePasswordRealm.java:71) ~[graylog.jar:?]
	at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:571) ~[graylog.jar:?]
	at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doMultiRealmAuthentication(ModularRealmAuthenticator.java:225) ~[graylog.jar:?]
	at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:275) ~[graylog.jar:?]
	at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198) ~[graylog.jar:?]
	at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106) ~[graylog.jar:?]
	at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:275) ~[graylog.jar:?]
	at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:260) ~[graylog.jar:?]
	at org.graylog2.shared.security.SessionCreator.create(SessionCreator.java:82) ~[graylog.jar:?]
	at org.graylog2.rest.resources.system.SessionsResource.newSession(SessionsResource.java:142) ~[graylog.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_312]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_312]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_312]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_312]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) [graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) [graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [graylog.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) [graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) [graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) [graylog.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [graylog.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) [graylog.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2021-12-16T13:18:24.510-05:00 INFO  [SessionCreator] Invalid credentials in session create request. Actor: "urn:graylog:user:joe_user"

tmacgbay · December 16, 2021, 10:00pm

Hmmm I have seen a few things on how password_secret in server.conf is used for salting passwords… is that something that might have changed in your test configuration or maybe wonked in translation?

mntbighker · December 16, 2021, 10:19pm

No, the hash has not changed. Non-LDAP logins work just fine.

gsmith · December 17, 2021, 3:26am

Hello

I found this. Different version looks like same issue.

mntbighker · December 17, 2021, 5:03am

I don’t have illegal key size, so not the same issue. Seems like the root cause is No such provider: SunJCE. I did more testing with a different AD account, same issue. Tried with TLS, same issue.

mntbighker · December 17, 2021, 5:18am

OK, so I’m a nitwit. We had an audit last week and enabled FIPS mode on all Linux systems. I’m REALLY confident this is what killed LDAP login. Probably SunJCE can’t function in that cryptographic mode. I have not verified, but I’m pretty sure that is the cause. Of course there is no reference to FIPS in Graylog documentation.

Topic		Replies	Views
New Install - AD Failing Graylog Central (peer support)	14	2056	October 6, 2021
LDAPConnector trying to connect to LDAP Server null Graylog Central (peer support)	2	596	October 17, 2019
LDAP authentication failed Development windows	6	818	January 18, 2024
LDAP not working Graylog Central (peer support)	9	2836	March 20, 2018
Cannot connect to LDAPS Graylog Central (peer support)	10	2304	March 4, 2022

LDAP auth broken with recent patch? FIPS mode breaks openjdk crypto

Related topics