Keycloak.X Boostrapping Admin Access #8549

pedroigor · 2021-10-08T14:20:56Z

pedroigor
Oct 8, 2021
Collaborator

We should provide a better experience for managing admin access to a Keycloak server. There are two main use cases we want to support, as suggested by @stianst :

Create the initial admin account
Recovering access if you lose the admin password

The corresponding operations to support the aforementioned use cases should rely on specific commands available from the CLI.

Create the initial admin account

This operation should be as simple as creating the master admin account for the first time. Helpful to automate how servers are preovisioned.

Recovering access when admin account is lost

This operation should allow the password to be recovered. There are some things to consider here like how to securely give information about the master admin.

Perhaps we could rely on sending an e-mail based on the e-mail previously set to the admin account. If not set, recovery should not be possible.

Or perhaps we could generate recovery codes or basic support for 2FA.

pedroigor · 2021-10-18T15:34:35Z

pedroigor
Oct 18, 2021
Collaborator Author

I can think about two possible approaches to this this problem:

As suggested by @stianst, leverage kcadm.sh to manage admin access
Provide specific commands in Dist.X CLI (kc.sh) to manage admin access

In both cases, users should be created based on a local-secure-random token (probably created when the server is started for the first time) so that access is restricted to localhost.

I think both options are mutually exclusive, and I'm not sure which one is the best.

By leveraging kcadm.sh, we are actually providing much more possibilities than just managing admin access but also complete access to our Admin API. That is a plus because users can also build their automation based on our Admin API. Also, we probably want to make kcadm.sh installation/usage easier when running at localhost, regardless.

On the other hand, providing a specific command in Dist.X CLI would provide a simpler experience to solve this particular problem?

9 replies

vmuzikar Oct 19, 2021
Collaborator

@jonathanvila This does not concern standard users, only the admin user in master realm, I'd say.

pedroigor Oct 19, 2021
Collaborator Author

Regarding the usage of the scripts execution....this involves doing ssh to the pod, so the Keycloak Admin would also be someone with access to the cluster and rights to mess with the installation, as it can ssh into a pod. .... hummmm I'm not sure.

I also do not think as insecure if we go through the kcadm.sh route. The reason is the same as yours. It is trade-off really, it should make provisioning a lot easier to do because in addition to creating the admin user you could create realms. All that without having to deal with OAuth2 grant types to first obtain a token and use it as a bearer. The script would be ready to use.

But yeah, I do not like kcadm.sh route but having a more specific/simpler command in our CLI. I won't touch kcadm.sh for now. It also deserves some more love in order to improve its UX (mainly due to how hard is to use our representations when posting payloads to the server).

jonathanvila Oct 19, 2021

@jonathanvila This does not concern standard users, only the admin user in master realm, I'd say.

yep I know....admin user in Keycloak, not a K8s admin user is what I mean.

pedroigor Oct 19, 2021
Collaborator Author

Also, how would clustering work? Each instance would have a unique token?

Do we need clustering at all? The driven use case here is provisioning as it should be enough to have the token bound to a specific instance only.

As I mentioned before, this token should be shared by the CLI and the server itself. So users won't need anything (no copy/paste, not bearer, no token exchange) else but just run the command to create the admin user.

stianst Oct 25, 2021
Maintainer

We need to consider also how an admin user is bootstrapped in the container and operator. Often that needs to be done without access to the pod directly. One thing worth mentioning is that we can use environment variables and secrets for the initial creation, but should not use that once it's bootstrapped.

vmuzikar · 2021-10-19T10:25:40Z

vmuzikar
Oct 19, 2021
Collaborator

We need to keep the cloud env in mind for the initial admin creation (and maybe even for the restoration). Even though the Operator.X won't need it (it won't be managing any KC resources), users need to be able to simply login to their new KC instances. And SSHing to the pod (to execute bunch of commands locally) is probably not the most user friendly solution.

We should probably allow creating an admin user from a file that would be actually a mounted K8s secret (env vars are considered less secure).

The question is, what about restoring access? What if the secret is then modified by the user? Do we care? Will it take any effect in KC? Should we even allow changing the admins password via REST API, or should we just rely on file/secret?

2 replies

pedroigor Oct 19, 2021
Collaborator Author

I like the idea of having admin created based on mounted secrets and it should be possible already when using the keycloak-add-user.json file. Perhaps I missed this capability in my options list. So, shall we keep the same behavior for Dist.X? If so, I would say that we could make it simpler when reading as a mounted secret.

To be honest, I'm not 100% sure about restoring access. Two reasons here:

Is that something we really need to care about considering today we don't have anything similar? How people are dealing with this today?
Once the admin user is created, people should be able to follow the available options to recover the account if they have properly configured the admin account as such. Including support for (M|2)FA.

W.r.t. to admin user creation, I think the tool should allow creating the admin user only once and deny any subsequent try to do the same.

jonathanvila Oct 19, 2021

What I see with a recovery system that implies to go into internal objects in the cluster is that we are expecting that the Keycloak admin user can also log into the cluster and touch things.... I am not sure if that's the case.
I think we are talking a business feature ( pass recovery ) , similar to the one in google or gmail....and I dont have to know or log into the google internals.

stianst · 2021-10-25T08:41:12Z

stianst
Oct 25, 2021
Maintainer

Summarising use-case:

Developers runs Keycloak locally and wants to login as admin
Production needs to bootstrap the first user for an initial setup (ZIP, container, and operator)
Production needs to bootstrap a service account instead of a user
Ability to recover access if admin passwords are lost

Not really convinced the current approach we have with kc-add-user + environment variables are the best, especially on the container and operator. Leeds to questions like should the admin password be updated if the env variable changes, and things like the operator keeping a secret with the admin password (that results in sharing an admin account).

Some thoughts on perhaps a different way:

For dev-mode, why not just have a default username/password? Or, the local authentication mode to kcadmin?
For production mode, why not generate a token that is printed to the log, and/or stored in a secret. This can be used to bootstrap the first account through the UI, or CLI

Not sure about recover admin access, but if you think about anyone that has access to the KC process and server config files will be able to get access to the DB secret (even if stored in a vault). So, the local auth for kcadmin could make sense, or perhaps just connect to the DB directly, with a special way to start the server like we do today.

In essence though I'd like to see a bit more UX-centric discussion happening here.

5 replies

vmuzikar Oct 26, 2021
Collaborator

+1 for having some default creds in dev mode.
I'd go with the path of reading admin creds just from a file (could be locally generated through kc.sh – pretty much same as now). That could work for all dist types (ZIP, container, operator) as Secrets mounted as env vars are considered less secure than a volume mount. On the other hand, we can expect that users will change the admin creds after the initial bootstrap so the original creds won't work. I believe we can even expect that the users adds additional layer of security to master admin accounts, like OTP or WebAuthn. For the same reason, it IMHO doesn't make much sense for the container/operator to react on the Secret changes. That would work only during first boot.
Isn't this a bit different use case, unrelated to the initial admin user? We always need an admin user, don't we?
Do we really need to care about this in container/operator dist types? Is it such a common use case users get locked out? Shouldn't we rather rely on the standard recovery flows (password reset link, recovery codes etc.). In any case, kc.sh should probably be able to reset the admin creds as a last resort, but that would be mainly for dev purposes I think.

bjolivot Nov 4, 2021

Please, no default cred. How many project did you saw moving directly from dev to prod ?
File based password is probably the more flexible way as it could be managed in classic server, K8S secret, any automation tool...

bs-matil Jan 3, 2022

Actually I like the idea of an one off token, as it handles the secret as ephemeral as any access is handled elsewhere (e.g. key rotation patterns with vaults) . Also this may could be combined with an SPI to publish the secret to k8s/vault/etc so surrounding infrastructure can pick it up, actually something which can maybe aligned with the client credential handling. For example an operator could expect a secret with the token, while keycloak boots it creates the secret and the operator initializes itself.

I have to think about this further but somehow this is appealing.

pedroigor Jan 3, 2022
Collaborator Author

@bs-matil Sounds appealing, but do we really need that level of integration between Keycloak and the underlying stack?

I guess this also means additional configuration options to connect the server with k8s API server?

bs-matil Jan 3, 2022

@pedroigor yes it would mean that. But as a note on that, k8s provides plenty of options to do a self discovery here (like a fixed way to provide access to the internal API, RBAC standards etc.) so it would be possible to provide sane defaults for this mechanism.

Nevertheless I am aware of the possibility to over-engineer this. On the other hand we could have a simple file system implementation which just publishes the token as file to integrate with classic approaches and enable such a behaviour specifically for k8s and the operator model.
A different question is still what is provided after the "publish SPI" is called. A token or a user.

Leletir · 2022-01-03T15:12:08Z

Leletir
Jan 3, 2022

Hi, as discussed with @vmuzikar in discussion 9362, I have the following use case (which is the same as reading env vars from mounted volume IMHO):

Admin credentials are stored in Hashicorp Vault
We use Hashicorp Vault Injector to fetch our secrets from our vault to our pods.
Thanks to the Vault Injector we obtain a file under /vault/secrets, for example:

/vault/secrets/admin-credentials

KEYCLOAK_ADMIN=toto
KEYCLOAK_ADMIN_PASSWORD=test

I would like to know if it's possible to read those values at boot time ?

1 reply

pedroigor Jan 3, 2022
Collaborator Author

It should be possible and we should be able to leverage the current support for mounted secrets or even Hashicorp. But not implemented.

In general, I think we all agree about leveraging our Vault SPI to fetch admin credentials? And that is probably the best/recommended approach for setting up admin access when running on k8s?

After re-reading the discussions here, I'm not sure if we need to change the welcome page and how people create the admin account. I mean, the UX is not bad, it is simple, and it works in case you are running in dev mode or can't/don't want to use secrets from Vault. I also agree with @bjolivot about no default cred.

michelefa1988 · 2022-04-18T16:47:35Z

michelefa1988
Apr 18, 2022

Hi Guys,

What is the latest about this? I would like to mount the KEYCLOAK_ADMIN_PASSWORD from a file but I am not sure how to import it. The password is already mounted in /secret/keycloak/KEYCLOAK_ADMIN_PASSWORD but not sure how to get the image to create the actual username?

Thanks

0 replies

pedroigor · 2022-05-24T19:08:11Z

pedroigor
May 24, 2022
Collaborator Author

After reviewing this discussion with @stianst, it is more clear the use cases we want to support:

Bootstrapping
Recovering Access
Automation

Bootstrapping

This use case is about starting the server for the first time and gaining access to the server to perform management operations.

Currently, the master admin account is created by:

Setting environment variables
Through the welcome page that is only accessible through the loopback interface
Import Realm

Using environment variables

Creating the master admin user using environment variables is not secure and considered a bad practice. Credentials are in plain text and can easily be accessed if you have access to the environment/container where the server is running. For instance, one can easily see the credentials from using a CLI or the UI if using Openshift.

An alternative to environment variables is to use mounted secrets but they are still open from unwanted access if you have access to the environment where the server is deployed.

Note that this mechanism is mainly useful when you can't access the welcome page to create the admin user because you can't access the server through the loopback interface (running as a container) or when you want automation happening right after the server is started to provision realms.

Welcome Page

Creating users from the welcome page should be OK and kept as is. However, it does not help much if you need automation right after the server is started, like provisioning realms.

Import Realm

If you are fluent enough to create a JSON file with realm representations, importing a realm at startup can help to not only bootstrap the master admin user but also any other data you need to provision.

This approach does not provide the necessary flexibility in terms of configuration because the realm configuration is statically defined in a JSON file.

Recovering Access

The solution for recovering access is not so clear. Here we can use recover links or just specific commands in the CLI to manage admin access as previously suggested.

Other than that, recovering access for admin accounts should leverage the existing capabilities provided by the server like using email and action tokens.

Automation

The way we are doing automation is wrong because it relies on the master admin user. We should favor instead using master service accounts for a number of reasons, some of them are:

It avoids unwanted access via the administration console because service accounts can not log in to the server
Automation is really about using clients and service-to-service communication. Users should not be involved at all.
By using a client there are a number of options to securely authenticate to the server thus leveraging the available client authentication methods such as mTLS, JWT authentication, etc. Including support for key rotation.

If we create a master service account when bootstrapping the server, as previously mentioned, we should be able to provide a better and more secure ground for automation.

If we go through the master token path, we would need something else to provision the service account like using the CLI.

Proposal

We keep the welcome page and import realm methods for bootstrapping the master admin user and realms.

We drop support for environment variables to create the master admin user. We also do not support mounted secrets. I know that has a deep impact on UX and how you are using containers and the operator, but at the same time relying on environment variables or mounted secrets is not really secure. Perhaps we can still support them with a trade-off and I would like to hear what others think about it.

Instead, we should consider introducing a master token that has enough entropy and is generated when the server is started for the very first time. Users should make sure they keep the token secure. The token itself can be used to access the Admin REST API for automation purposes. This token should never be persisted and used only for the initial setup like provisioning realms and their data.

The problem with the master token is that we need additional barriers around it to make it secure and hard to generate. For instance, it should be fine to log the token in logs when running in dev mode but when at production we should be able to generate this token using the CLI and that means an additional step prior to doing any automation. We should also make sure that any access using this token is audited (e.g.: using event types/listeners).

If the proposal around the master token is accepted, I'm open to creating a design document for further discussions.

By introducing the master token concept we should also be able to provide additional commands in our CLI that are specific for managing admin access, similar to the AddUser CLI available in the legacy distribution.

An alternative to the master token is creating a master-service-account client and exposing its secrets during startup, and potentially providing key materials at startup to enable a more secure client authentication method other than client secret.

5 replies

vmuzikar May 25, 2022
Collaborator

@pedroigor Thanks for the elaborate proposal!

I'd like to discuss a bit more the cloud-native UX here (without going too much into implementational details for now).

I believe we could certainly default to master token. The Operator would make sure to create a Secret with the generated token. However, I'm not sure if it's a great UX not to have the option to create some standard initial admin user as well. I think this really boils down to our broader cloud-native vision of Keycloak management. The way I see it we have two possible approaches:

No Admin Console
- The Admin Console would be disabled by default.
- We would offer only master token without creating initial admin user.
- It would be possible to configure anything in Keycloak in cloud-native way without the need for Admin Console, i.e. we'd rely on CRs.
Optional Admin Console
- We would still default to option 1 above but in addition to it, it'd be possible to explicitly enable Admin Console access using initial admin user.
- Admin Console would be enabled.
- Initial admin user would be created and credentials stored in a Secret.
- Only selected areas would be configurable through CRs.

My point is that IMHO if we didn't create initial admin user and we encouraged people to use service accounts (and master token) instead, we would not expose Admin Console by default at all and we would put more emphasis on cloud-native config (CRs) capabilities.

However, I think we still have more implementational options how to avoid using env vars our mounting Secrets. The initial admin credentials could be generated by Keycloak (using a special CLI command). The Operator would run a one-time Job to generate and retrieve the credentials from Keycloak and it'd then save it into a Secret.

pedroigor May 25, 2022
Collaborator Author

Thanks.

I'm not sure if relying on CRs is feasible until we completely review our representations. Perhaps if we decide that CRs are going to be a wrapper for our current representations and provide a better structure for managing realms.

However, I think we still have more implementational options how to avoid using env vars our mounting Secrets. The initial admin credentials could be generated by Keycloak (using a special CLI command). The Operator would run a one-time Job to generate and retrieve the credentials from Keycloak and it'd then save it into a Secret.

Or the operator can generate a master token/service account using a job and use it to provision users and service accounts through REST and no need for additional commands in our CLI.

Btw, is the operator using the master admin user to access the Admin REST API?

vmuzikar May 25, 2022
Collaborator

I'm not sure if relying on CRs is feasible until we completely review our representations.

+1000
This is what I forgot to mention. For us to rely on CRs, we need the static store. And the static store needs revamping our current representations, if I understood correctly.

Btw, is the operator using the master admin user to access the Admin REST API?

No, the new Operator doesn't access the REST API at all. And I think we should stick with that. Manipulating REST API brings additional complexity and even potential security vulnerabilities.

syats Dec 28, 2022

I think the master_token as proposed is almost interchangeable with setting an admin password using env, if the operator is forced to changed the password at first login (an already supported policy). Yes, someone with access to the running container could inspect the password, but this would have already been changed.

In any case, if implementing the master_token approach, please do not remove the option to set admin password via environment variables, as this is standard procedure in many other systems (e.g. DBMSs supported by keycloak).

andre-nascimento6791 Jun 7, 2023

Great Job in elaborating this Proposal, @pedroigor and Good Catch, @vmuzikar !

I agree to the option of going for something like a master token, but also providing the two possible approaches (No Admin Console and Optional Admin Console) as wisely suggested by @vmuzikar.

We are aware that there are others issues related to kcadm.sh | kcadm.bat CLI tool. It needs some fixes and improvements, and probably a re-design effort. After version 22 release, maybe we could come back to this topic (Bootstraping Admin Access / Support for Admin recovery) and consider each need in both design efforts.

thomas-riccardi · 2022-06-09T17:13:55Z

thomas-riccardi
Jun 9, 2022

The current recovery story is a pain:

Before upgrading to recent keycloak, our admins used direct access to keycloak processes to create a new, temporary, admin user and password with the environment variables (via kubernetes/helm).
This is now broken when there are already other users in the master realm, according to #12169.

We tried to simply delete all users in the master realm via raw SQL, but it's hard to do as there is no cascade delete configured...

For what it's worth, our current procedure is to manually created via raw SQL a credential password entry for an existing admin master user, with the following generation script:

ADMIN_USER_ID=09fd95c9-b774-4c87-a9f1-17f58e60c4da
PASSWORD=$(pwgen -s 22 1)
python3 - <<OEF
import os
from base64 import b64encode
from hashlib import pbkdf2_hmac
import json
import uuid

salt = os.urandom(16)
iterations = 27500
secret_value = pbkdf2_hmac('sha256', b'$PASSWORD', salt, iterations, dklen=64)

secret_data=json.dumps({"value": b64encode(secret_value).decode(), "salt": b64encode(salt).decode()})
credential_data=json.dumps({"hashIterations": iterations, "algorithm": "pbkdf2-sha256"})

print(f"INSERT INTO credential (id, type, user_id, secret_data, credential_data, priority) VALUES ('{uuid.uuid4()}', 'password', '$ADMIN_USER_ID', '{secret_data}', '{credential_data}', 10)")
OEF
echo Use this password to login: $PASSWORD

(It seems the cache sometimes causes issues and the login doesn't work; a restart helps.)

=> It would be great to have a much easier way to recover access via server access like we had before.

0 replies

rezaokhravi · 2023-01-31T08:29:25Z

rezaokhravi
Jan 31, 2023

Hello, I have an error in the following code, can you help me?

version: '3.6'
services:
  mssql:
    container_name: mssql
    image: mcr.microsoft.com/mssql/server:2019-latest
    restart: always
    ports:
      - 1430:1433
    environment:
      - ACCEPT_EULA=Y
      - MSSQL_PID=Enterprise
      - MSSQL_SA_PASSWORD=Aa123456
    volumes:
      - ./volumeMssql:/var/opt/mssql/data
  mssqlscripts:
    container_name: mssqlscripts
    image: mcr.microsoft.com/mssql-tools
    depends_on:
      - mssql
    command: /bin/bash -c 'until /opt/mssql-tools/bin/sqlcmd -S mssql -U sa -P "Aa123456" -Q "create database Keycloak"; do sleep 5; done'
  keyclock:
    container_name: keyclock
    image: quay.io/keycloak/keycloak:20.0.3
    restart: always
    depends_on:
      - mssql
      - mssqlscripts
    ports:
      - 8080:8080
    environment:
      - KEYCLOAK_ADMIN=admin
      - KEYCLOAK_ADMIN_PASSWORD=Aa123456
      - KC_HEALTH_ENABLED=true
      - KC_METRICS_ENABLED=true
      - KC_DB=mssql
      - KC_DB_URL=jdbc:sqlserver://mssql:1430;databaseName=Keycloak
      - KC_DB_USERNAME=sa
      - KC_DB_PASSWORD=Aa123456
    entrypoint: [ "/opt/keycloak/bin/kc.sh", "start-dev" ]

0 replies

lujiefsi · 2023-04-10T16:41:29Z

lujiefsi
Apr 10, 2023

hi recently, i report a issue by
https://huntr.dev/bounties/48ff1281-addd-49e4-b457-d8d4bb3599d7/

now keycloak can delete all users,include himself. i think we shoud add a check to prevent a user can delete himself

0 replies

abhishekranjan1 · 2024-02-28T02:08:58Z

abhishekranjan1
Feb 28, 2024

I am trying to run the following -
docker run --name mykeycloak -p 8080:8080
-e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=change_me
quay.io/keycloak/keycloak:latest
start
--db=postgres --features=token-exchange
--db-url= --db-username= --db-password=
--https-key-store-file= --https-key-store-password=

But getting an error that says -
Unknown option: '--features'
Try 'kc.sh --help' for more information on the available options.

1 reply

abhishekranjan1 Feb 28, 2024

I followed the instructions from here -https://www.keycloak.org/server/containers

GreySpike · 2024-05-14T10:43:52Z

GreySpike
May 14, 2024

Importing on startup is problematic when there are multiple files because no order is defined.

In my case I wanted to separate the definition of a new realm with the definition of the users in that realm, so that QA could easily add more users without changing the realm, but ExportImportManager does not sort the file list (line 124) so it happened that the users were imported first before the realm was defined.

I might try to make the changes myself, but I am not a sw developer (well not since the early 1990's with Java 0.3)

0 replies

vmuzikar · 2024-05-14T11:15:00Z

vmuzikar
May 14, 2024
Collaborator

Created a summarized design proposal here. Input would be most appreciated.

0 replies

Keycloak.X Boostrapping Admin Access #8549

Uh oh!

pedroigor Oct 8, 2021 Collaborator

Create the initial admin account

Recovering access when admin account is lost

Replies: 12 comments · 23 replies

Uh oh!

pedroigor Oct 18, 2021 Collaborator Author

Uh oh!

vmuzikar Oct 19, 2021 Collaborator

Uh oh!

pedroigor Oct 19, 2021 Collaborator Author

Uh oh!

Uh oh!

pedroigor Oct 19, 2021 Collaborator Author

Uh oh!

stianst Oct 25, 2021 Maintainer

Uh oh!

vmuzikar Oct 19, 2021 Collaborator

Uh oh!

pedroigor Oct 19, 2021 Collaborator Author

Uh oh!

Uh oh!

stianst Oct 25, 2021 Maintainer

Uh oh!

Uh oh!

vmuzikar Oct 26, 2021 Collaborator

Uh oh!

Uh oh!

Uh oh!

pedroigor Jan 3, 2022 Collaborator Author

Uh oh!

Uh oh!

Uh oh!

pedroigor Jan 3, 2022 Collaborator Author

Uh oh!

Uh oh!

Uh oh!

pedroigor May 24, 2022 Collaborator Author

Bootstrapping

Using environment variables

Welcome Page

Import Realm

Recovering Access

Automation

Proposal

Uh oh!

Uh oh!

vmuzikar May 25, 2022 Collaborator

Uh oh!

pedroigor May 25, 2022 Collaborator Author

Uh oh!

vmuzikar May 25, 2022 Collaborator

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pedroigor
Oct 8, 2021
Collaborator

Replies: 12 comments 23 replies

pedroigor
Oct 18, 2021
Collaborator Author

vmuzikar Oct 19, 2021
Collaborator

pedroigor Oct 19, 2021
Collaborator Author

pedroigor Oct 19, 2021
Collaborator Author

stianst Oct 25, 2021
Maintainer

vmuzikar
Oct 19, 2021
Collaborator

pedroigor Oct 19, 2021
Collaborator Author

stianst
Oct 25, 2021
Maintainer

vmuzikar Oct 26, 2021
Collaborator

pedroigor Jan 3, 2022
Collaborator Author

pedroigor Jan 3, 2022
Collaborator Author

pedroigor
May 24, 2022
Collaborator Author

vmuzikar May 25, 2022
Collaborator

pedroigor May 25, 2022
Collaborator Author

vmuzikar May 25, 2022
Collaborator