Before reporting an issue

• I have searched existing issues
• I have reproduced the issue with the latest nightly release

Area

core

Describe the bug

Issue with FIPS140 in non approved mode.

When starting KC 22 in non approved mode(when following the doc guide), with FIPS configured, it generates:

bin/kc.sh start --features=fips --hostname=localhost --https-key-store-password=passwordpassword --log-level=INFO,org.keycloak.common.crypto:TRACE,org.keycloak.crypto:TRACE

...
....

023-09-07 08:27:29,386 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (production) mode
2023-09-07 08:27:29,386 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Unable to start HTTP server
2023-09-07 08:27:29,386 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: java.io.IOException: Invalid keystore format
2023-09-07 08:27:29,386 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Invalid keystore format
...

Version

22.0.1

Expected behavior

Following command provided in the doc should work

reference used: FIPS guide
https://www.keycloak.org/server/fips

Actual behavior

bin/kc.sh start --features=fips --hostname=localhost --https-key-store-password=passwordpassword --log-level=INFO,org.keycloak.common.crypto:TRACE,org.keycloak.crypto:TRACE

...
....

023-09-07 08:27:29,386 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (production) mode
2023-09-07 08:27:29,386 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Unable to start HTTP server
2023-09-07 08:27:29,386 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: java.io.IOException: Invalid keystore format
2023-09-07 08:27:29,386 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Invalid keystore format
...

How to Reproduce?

1. Reference:
   https://www.keycloak.org/server/fips

2. INFRA used:

RHEL 9 with FIPS enabled
OpenJDK 17 installed

$ fips-mode-setup --check
FIPS mode is enabled.

3. Prepare Bouncy castle library

ls $KEYCLOAK_HOME/providers
README.md bc-fips-1.0.2.3.jar bcpkix-fips-1.0.7.jar bctls-fips-1.0.14.jar

4. create cat kc.keystore-create.java.security

cat kc.keystore-create.java.security
securerandom.strongAlgorithms=PKCS11:SunPKCS11-NSS-FIPS

5. generate keystore

$ keytool -keystore $KEYCLOAK_HOME/conf/server.keystore -storetype bcfks -providername BCFIPS -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath $KEYCLOAK_HOME/providers/bc-fips-*.jar -alias localhost -genkeypair -sigalg SHA512withRSA -keyalg RSA -storepass passwordpassword -dname CN=localhost -keypass passwordpassword -J-Djava.security.properties=/tmp/kc.keystore-create.java.security

Generating 2,048 bit RSA key pair and self-signed certificate (SHA512withRSA) with a validity of 90 days

6. Keystore verification

$ keytool -keystore $KEYCLOAK_HOME/conf/server.keystore -storetype bcfks -providername BCFIPS -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath $KEYCLOAK_HOME/providers/bc-fips-*.jar -list -v -storepass passwordpassword -J-Djava.security.properties=/tmp/kc.keystore-create.java.security
Keystore type: BCFKS
Keystore provider: BCFIPS

Your keystore contains 1 entry

Alias name: localhost
Creation date: Sep 7, 2023
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=localhost
Issuer: CN=localhost
Serial number: a8a8594aa6e509f0
Valid from: Thu Sep 07 08:34:23 EDT 2023 until: Wed Dec 06 07:34:23 EST 2023
Certificate fingerprints:
SHA1: 7A:4D:D0:5F:00:73:C9:AF:83:5B:FA:76:B1:B1:A3:09:90:AF:3F:B0
SHA256: 84:68:DF:4E:D8:87:3B:41:0D:DB:ED:E5:48:4C:1F:7F:E9:DB:02:B9:BE:A9:EB:BF:87:88:38:61:07:76:62:C4
Signature algorithm name: SHA512WITHRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

7. testing in approved mode

bin/kc.sh start --features=fips --hostname=localhost --https-key-store-password=passwordpassword --log-level=INFO,org.keycloak.common.crypto:TRACE,org.keycloak.crypto:TRACE

2023-09-07 08:27:22,803 WARN [org.keycloak.common.util.KerberosJdkProvider] (main) Kerberos feature not supported by JDK. Check security providers for your JDK in java.security. Supported mechanisms: []
2023-09-07 08:27:23,678 DEBUG [org.keycloak.crypto.fips.FIPS1402Provider] (main) Strong secure random not available. Tried algorithms: NativePRNGBlocking:SUN,DRBG:SUN. Using algorithm as a fallback for strong secure random: PKCS11:SunPKCS11-NSS-FIPS
2023-09-07 08:27:23,685 DEBUG [org.keycloak.crypto.fips.FIPS1402Provider] (main) Initialized BCFIPS secured random
2023-09-07 08:27:23,689 DEBUG [org.keycloak.crypto.fips.FIPS1402Provider] (main) Inserted security providers: [BCFIPS, BCJSSE]
2023-09-07 08:27:23,689 DEBUG [org.keycloak.common.crypto.CryptoIntegration] (main) Using the crypto provider: org.keycloak.crypto.fips.FIPS1402Provider
2023-09-07 08:27:23,831 INFO [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: , Hostname: localhost, Strict HTTPS: true, Path: , Strict BackChannel: false, Admin URL: , Admin: , Port: -1, Proxied: false
2023-09-07 08:27:24,624 TRACE [org.keycloak.common.crypto.CryptoIntegration] (main) Java security providers: [
KC(BCFIPS version 1.000203, FIPS-JVM: enabled) version 1.0 - class org.keycloak.crypto.fips.KeycloakFipsSecurityProvider,
BCFIPS version 1.000203 - class org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider,
BCJSSE version 1.0014 - class org.bouncycastle.jsse.provider.BouncyCastleJsseProvider,
SunPKCS11-NSS-FIPS version 17 - class sun.security.pkcs11.SunPKCS11,
SUN version 17 - class sun.security.provider.Sun,
SunEC version 17 - class sun.security.ec.SunEC,
SunJSSE version 17 - class sun.security.ssl.SunJSSE,
SunJCE version 17 - class com.sun.crypto.provider.SunJCE,
SunRsaSign version 17 - class sun.security.rsa.SunRsaSign,
XMLDSig version 17 - class org.jcp.xml.dsig.internal.dom.XMLDSigRI,
]
2023-09-07 08:27:24,624 TRACE [org.keycloak.common.crypto.CryptoIntegration] (main) Security properties: [
Java security properties file: null
Default keystore type: pkcs12
KeyManagerFactory.getDefaultAlgorithm(): X.509
TrustManagerFactory.getDefaultAlgorithm(): PKIX
keystore.type.compat: true
javax.net.ssl.trustStoreType: pkcs12
javax.net.ssl.trustStore: null
javax.net.ssl.trustStoreProvider: null
javax.net.ssl.keyStoreType: null
javax.net.ssl.keyStore: null
javax.net.ssl.keyStoreProvider: null
]
2023-09-07 08:27:25,233 WARN [io.quarkus.agroal.runtime.DataSources] (main) Datasource enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
2023-09-07 08:27:25,352 DEBUG [org.keycloak.crypto.fips.KeycloakFipsSecurityProvider] (agroal-11) Returning DEFAULT algorithm of BCFIPS provider instead of SHA1PRNG
2023-09-07 08:27:25,353 DEBUG [org.keycloak.crypto.fips.KeycloakFipsSecurityProvider] (Generate Seed) Returning DEFAULT algorithm of BCFIPS provider instead of SHA1PRNG
2023-09-07 08:27:25,671 WARN [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2023-09-07 08:27:25,814 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2023-09-07 08:27:25,942 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
2023-09-07 08:27:26,100 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel ISPN
2023-09-07 08:27:26,106 INFO [org.jgroups.JChannel] (keycloak-cache-init) local_addr: 74b7183d-e02c-47fe-8a06-478d0cd0ba1b, name: rhbk-1-ouser1-7551
2023-09-07 08:27:26,111 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB
2023-09-07 08:27:26,112 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 20MB, but the OS only allocated 212.99KB
2023-09-07 08:27:26,112 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB
2023-09-07 08:27:26,115 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB
2023-09-07 08:27:26,144 INFO [org.jgroups.protocols.FD_SOCK2] (keycloak-cache-init) server listening on *.31305
2023-09-07 08:27:26,649 INFO [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2023-09-07 08:27:28,166 INFO [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) rhbk-1-ouser1-7551: no members discovered after 2011 ms: creating cluster as coordinator
2023-09-07 08:27:28,172 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [rhbk-1-ouser1-7551|0] (1) [rhbk-1-ouser1-7551]
2023-09-07 08:27:28,175 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel ISPN local address is rhbk-1-ouser1-7551, physical addresses are [10.0.76.12:46841]
2023-09-07 08:27:28,472 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: rhbk-1-ouser1-7551, Site name: null
2023-09-07 08:27:29,362 INFO [org.infinispan.CLUSTER] (main) ISPN000080: Disconnecting JGroups channel ISPN
2023-09-07 08:27:29,386 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (production) mode
2023-09-07 08:27:29,386 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Unable to start HTTP server
2023-09-07 08:27:29,386 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: java.io.IOException: Invalid keystore format
2023-09-07 08:27:29,386 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Invalid keystore format
2023-09-07 08:27:29,386 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.

Anything else?

N/A