Did your changes also improve the performance for managing large numbers of clients (> 10000)?

It would need some measures here, probably. Are you referring to an existing issue? Is it just being able to create 10k+ clients in a single realm, or spread across multiple realms ? Admin console loading? Something else?
I'm happy to have a go and check if there is an impact of the changes to this scenario, just let me know.

Not in particular, but there are (were) several issues with realms that needs to manage a large number of clients, e.g.:
KEYCLOAK-8275, https://keycloak.discourse.group/t/maximum-number-of-clients-applications-in-a-realm/461/15 and many other google groups discussions that mention this.

There are multiple scenarios:

• having a large number of realms with a small set of clients each (this should work already, with this fix) (SaaS Product Scenario)
• having a small number of realms with a large set of clients each (IDaaS for B2B Partners / IoT scenario)
• having a large number of realms with potentially a large set of clients each (General purpose IDaaS / B2C IoT scenario)

The largest number of clients in a realm I've seen so far were ~40k and with this the system was quite slow (even with some cache size tuning). Although it is possible to support an arbitrary number of clients via the client-storage and a custom org.keycloak.storage.client.ClientStorageProvider it would be great if Keycloak could handle large amounts of clients (100k+-1m+) well out of the box.

I'm also curious to learn what the folks of the Keycloak team think about your optimizations in the context of the new upcoming storage architecture.

Regarding the impact on capability to handled 100k/1m clients:

Just had a go with the code provided with KEYCLOAK-8275 against stock Keycloak 17.0, and after adjusting the code to latest codebase (a missing commit to finalize realm creation, deployment model change with Quarkus so now manually copied to the container and restart), it looks to me that it performs already pretty well (unless the testing methodology does not reflect the problem at hand - can't tell myself here).

It took 09:22 (562 seconds) to create 1 million clients into the testRealm.
Snapshot of the metrics it produces:

-- Timers ----------------------------------------------------------------------
client-registration
             count = 928284
         mean rate = 1719.01 calls/second
     1-minute rate = 1748.08 calls/second

The database indeed shows these clients:

keycloak=# select count(*) from client where realm_id = 'testRealm';
  count  
---------
 1000006
(1 row)

And finally, connecting to the admin console, selecting the testRealm then finally going to Clients is instantaneous, including next / previous page, as well as searching:

Looks like it was adressed prior to the optimisations I propose here ;-)

Can't remember when the improvements around large number of clients where done, but we did a lot of improvements around here and KC should scale easily enough to 100K+ clients now.

HI @laskasn
I did run the HA integration test, but the test scope looks pretty narrow.
And I'm be really more comfortable if I can check that I properly handled cache entries invalidation checks in the provided changes - and this cannot be tested with a single node.
Currently, I'm a bit at loss to understand which Quarkus-based distro configuration knobs I need to turn to have multiple containers (would be running behind a reverse proxy, no pb here) and run the script against this cluster.
If you can point me to some directions I could use, I'll run the same against a fully load balanced clustered Keycloak (I'd go for 3 nodes as there are situations you can't reproduce with 2 nodes).
Any idea?

Got it, here it goes.

# Increase the host Kernel parameters to allow proper JGroups performance
docker run -it --rm --privileged --pid=host justincormack/nsenter1 /bin/sh -c '\
  echo 26214400 > /proc/sys/net/core/rmem_max; \
  echo 26214400 > /proc/sys/net/core/wmem_max'

Then create the first node:

node=1
docker run -it --rm --name keycloak${node} \
  --network keycloak \
  --cap-add SYS_ADMIN \
  -p 808${node}:8080 \
  -p 878${node}:8787 \
  -p 899${node}:899${node} \
  -e KEYCLOAK_ADMIN="keycloak" \
  -e KEYCLOAK_ADMIN_PASSWORD="keycloak" \
  -e KC_DB="postgres" \
  -e KC_DB_URL="jdbc:postgresql://postgres-keycloak.keycloak:5432/keycloak" \
  -e KC_DB_USERNAME="postgres" \
  -e KC_DB_PASSWORD="postgres" \
  -e DEBUG="true" \
  -e DEBUG_PORT="*:8787" \
  -e JAVA_OPTS_APPEND="-Xmx1g \
    -Dcom.sun.management.jmxremote.port=899${node} -Dcom.sun.management.jmxremote.rmi.port=899${node} \
    -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false \
    -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname="$(hostname)" \
    -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/keycloak.hprof" \
  quay.io/keycloak/keycloak:18.0.0-SNAPSHOT start-dev \
  --log-level=INFO \
  --proxy=edge \
  --cache=ispn --cache-stack=udp

Wait for the schema to be created, then run the two others nodes, by running the same command in separate shells with node=2 and node=3.

Finally, run a nginx load balancer:

cat > ./nginx-keycloak.conf <<EOF
upstream keycloak {
  server keycloak1.keycloak:8080;
  server keycloak2.keycloak:8080;
  server keycloak3.keycloak:8080;
}

server {
    listen       8080;
    listen  [::]:8080;
    server_name  localhost;

    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Port  $server_port;

    location / {
        proxy_pass http://keycloak;
    }
}
EOF

# Create the container without running it
docker create --name nginx-keycloak \
  --network keycloak \
  -p 8080:8080 \
  nginx:1.21.6-alpine
# Copy the conf file
docker cp ./nginx-keycloak.conf nginx-keycloak:/etc/nginx/conf.d/default.conf
# Finally start it
docker start nginx-keycloak

You should be able to hit http://localhost:8080, logging properly and using the admin console as usual.

Note that the nginx here is setup here to use round-robin, so each request will hit the next backend - this probably defeats the cookie-based approach (unless session are clustered and replicated across nodes?), but I wanted to be in the worst-case scenario to uncover any potential issue.

I first manually checked that any change in realms and roles on one node (each node exposes its own port, so you can by-pass the load balancer here), which properly invalidates previously cached data in the other nodes (refresh shows the updated data on the other nodes).

I ran the realm creation scripts up to 3k realms, no error occurred, and the creation time graph is:

Same linear pattern, with an additional cost around 100 ms - not sure where it comes from. Maybe synchronous Infinispan invalidation across nodes (did not have a look at the caching layer definition here, so this is a wild guess)?

Login into the Admin console with 3k realms using the load balancer is obviously more costly: it takes around 110 seconds.
This is expected because the API requests are round-robin balanced across the 3 code-started nodes, so it basically populates the caches from the database 3 times in this case.
By-passing the load balancer and hitting a single node (similar to what would occur if using sticky session at the LB level), the load time is back to 50 seconds.

If you mean how you can use the tool, the steps in the readme file should help. If not, we can improve.

And I forgot about something very important. Before moving with this we should probably have @stianst, @mposolda, and @hmlnarik approvals too ...

If you mean how you can use the tool, the steps in the readme file should help. If not, we can improve.

I was more thinking about how to use the tool in this specific context (I already scanned the readme). :-)

And I forgot about something very important. Before moving with this we should probably have @stianst, @mposolda, and @hmlnarik approvals too ...

Sure, any step required from me to get their attention on this?

Regarding the tool, you can use the "dataset" provider. This provider allows you to call specific endpoints to automatically provision realms and their data. The endpoints accept some parameters to set the number of realms, clients, users, etc. We can always add more if needed.

W.r.t. to their attention, let's wait for their reply here. The data you provided is much more detailed than what I did in the past so I hope we have more input to justify the investiment.

@davoustp - I will soon be working on the Red Hat engineering side to validate the performance of the new store. For that we plan to use the keycloak-benchmark project Pedro Igor mentioned above, so contributions to that project or tests that build on top of it are very welcome to ensure the new store will work nicely.

Without getting ahead of the maintainers to reply, big changes in the existing JPA would require extensive reviews and testing to get them approved. With the new store having priority, it will be difficult to find the time.

On the other hand, for the changes that are server-spi level, there is a (short) window once Keycloak 18 has been released to adapt existing interfaces. I wonder if your changes break an existing API as you seem to only add methods.

Therefore, as @pedroigor suggested, I'd rather keep the changes small and one at a time.

I will soon be working on the Red Hat engineering side to validate the performance of the new store. For that we plan to use the keycloak-benchmark project Pedro Igor mentioned above, so contributions to that project or tests that build on top of it are very welcome to ensure the new store will work nicely.

Nice! For sure, will do.

Without getting ahead of the maintainers to reply, big changes in the existing JPA would require extensive reviews and testing to get them approved. With the new store having priority, it will be difficult to find the time.

Ok, I get that. The changes are not that big, really, but yes, I do get the concern.

On the other hand, for the changes that are server-spi level, there is a (short) window once Keycloak 18 has been released to adapt existing interfaces. I wonder if your changes break an existing API as you seem to only add methods.

That's correct. I added a few methods, without changing existing ones, and provided default implementations whenever possible to keep things simple for existing code.

Therefore, as @pedroigor suggested, I'd rather keep the changes small and one at a time.
Yep, I got that. Thx!

Yeah, the dataset is basically about provisioning. There are no metrics or reports from it.

However, the benchmark tool is based on Gatling and gives you more detailed information when checking runtime performance. Not sure if this tool applies to your case though because there we have test scenarios for authentication and a few Admin APIs.

Would it make sense to have some light-weight metrics to it (compared to doing a full-blown Gatling scenario)?

That would be awesome and should allow us to evaluate provisioning without introducing the HTTP layer with a focus on the server internals.

Hi @saguntumkar,
All the changes are in fork https://github.com/davoustp/keycloak/tree/KEYCLOAK-4593-investigations .
This allows to have a look and check what changed, but will require to be split in various PRs to make to the main branch, as @pedroigor suggested.
We're awaiting from main Keycloak folks to have a look and provide some directions.

Hi @davoustp ,
Interested to see your findings, I have a query, after enabling JMX by setting params in JAVA_OPTS_APPEND were you able to connect jmx port? I am facing issue with connection. If it got worked really appreciate your support, a separate discussion is started for this in thread #11503

Hi @mposolda
Thanks for the extensive feedback! 👍
Sorry for the response delay, I was busy elsewhere.

1. It will be good to "decompose" the work into multiple PRs. The ideal is as small and isolated PRs as possible, so we can discuss individual points in more details.

Indeed, that's the intent. As you mention in your conclusion, I will start splitting the code changes in separated, smaller PRs.
I'll need to do it gradually, since some changes do rely on other changes, so some PRs will be depending on other PRs.
I'll try to make it as simple as possible.

2. There are lots of new additions in the server-spi module in the model SPI. ...

Ok, got that.

3. You mentioned that all the tests are working for you. Which DB did you tested with? One point here is, that all DB changes need to be tested also with all the databases, which Keycloak currently supports (MySQL, MariaDB, PostgreSQL, Oracle, MSSQL, PostgresPlus). Here in RH, we have internal pipeline and we are able to run the functional tests with all the databases.

I ran all the integration tests against Postgres. I can probably run them against Oracle and SQL Svr (as we already do that in our own products and CI pipelines), but it probably make more sense to leverage your existing pipeline at RH. Obviously, I don't have access to this pipeline, so what is the best approach here? Just going with the PRs and awaiting for feedback when the PR has been tested in the pipeline?

4. I wonder if the optimizations you propose improve performance for all the databases or if there is some chance that some DBs may not be that optimal? I believe that this is not the case and performance benefit is for all the DBs, but not 100% sure about it.

I'm pretty sure that all databases will benefit from it, because the change is about how the tables are queried - switching from a 1+N querying pattern to a log(N) + k querying pattern (queries being much more simple and fast in addition to this - these are basically bulk lookups by PK, instead of complex joins).

4. For example I have some concerns about the queries like select compositerole from CompositeRoleEntity compositerole where compositerole.compositeId in :roleIds in the case when there is very big amount of records in the in clause (few thousands or so). Actually I am not even 100% sure if this work fine from the functionality perspective.

The SELECT ... WHERE x IN (list) is standard SQL and is supported by JPA and Hibernate so I'd be very surprised that we run into a problem here.
But you're totally correct in expecting issues with the number of items into the list of ids provided to the query: each DB engine has its own limitation in this respect. For example, Oracle is 1000 max, Postgres is signed smallint.MAX...
That's what I mention into the initial post:

the JPA implementation must also take care to limit the number of ids to bulk load using a select ... from ... where id in :ids JPA query (as Oracle and other engines do have limit - 1000 for Oracle)

This has been implemented into the code changes, for example at https://github.com/davoustp/keycloak/blob/23e06582ff3b81b29d0e0cdb84eae0f9bcdec5d6/model/jpa/src/main/java/org/keycloak/models/jpa/JpaRealmProvider.java#L532
The nb of items is here capped at 512 items (being a power of 2 to optimize against Hibernate in_clause_padding feature which minimizes the nb of cached statements for these cases).
Now that I have the list of supported DB engines, I can have a quick look for each to see what is their own limits re: the IN clause.

4. Which raises the more general question if we should have also some "functional" testing with the big amount of records to make sure that refactoring doesn't break any DB vendor in the environments with big amount of records in the DB?

Yes, that would be great, but I would be at loss to propose something here. I did use the keycloak-benchmark provisioning facility to check the overall performance improvement, but this is a far cry from a dedicated, purpose-built integration test...

5. Backwards incompatible changes in the server-spi are always a bit problematic. But fortunately, it seems that your work doesn't have lots of them - I noticed just something in the RoleUtils class. In some cases, we can justify backwards incompatible changes (if really needed) and write about them to migration guide, so I hope that this will be fine. Just FYI.

Yes, I tried to keep these to the very bare minimal - maybe it can be stripped down even more (will keep that in mind when splitting into individual PRs).

6. For the admin console, my vote is towards the refactoring of the "logic" itself rather than trying to optimize everything just at the model layer. ...

I fully agree. I didn't want to enter a full change against the UI in addition to the model/storage system, so that's why I went the optimisation route, to ensure that most showstoppers were gone from a scalability perspective.
I think loading a single realm (and related roles) instead of the full realm and role list would be a much better approach - even more with searching and paging.

Conclusion: I am personally not sure how to move forward. Hopefully we can have some feedback soon from the new store team if the changes in the model SPI (EG. especially new methods for retrieving list of IDs for particular objects and retrieve the list of objects by list of IDs etc) makes sense for the new store.

That's probably one the very first PRs I'll extract, so we can discuss it. For sure it would benefit the map-jpa implementation, and probably not so much for the key-value based map stores (in this case, a naive implementation performing the lookups would be enough from a performance perspective). But let's wait for the storage team to provide their feedback.

In parallel, we can rethink how to optimize the WhoAmI (my point 6) - I wonder it will probably makes sense to postpone once new admin console is the default as it will require some changes in the UI and will be nice to avoid doing them for both new and old admin console.

Yep, I think that makes sense to review these changes after the other model / storage PRs have been dealt with, especially if only the new UI is concerned and go for the single-realm loading approach.

Again, thanks for your feedback!

Hi, I'm joining into this discussion for the storage team, and I'm happy to review the PRs from the storage team side of things.

It's good to hear that you'll be testing PostgreSQL, and also look into restrictions that OracleDB would have. Once there is a PR and we see that it works for the GitHub actions, I can run it also against the internal pipeline for the other databases, please ping me for that.

Regarding the SPIs: there is a small window for Keycloak 19 and possibly 20 were a change could be done a little bit easier with regards to backward compatibility, still the earlier the better.

That being said, please ping me (@ahus1) on any PR you create to this. I'd also like to see a PR for the https://github.com/keycloak/keycloak-benchmark/tree/main/benchmark/src/main/scala/keycloak/scenario/admin with a scenario that could be similar to "CreateClients".

Hi @ahus1
Thanks for joining ;-)

It's good to hear that you'll be testing PostgreSQL, and also look into restrictions that OracleDB would have. Once there is a PR and we see that it works for the GitHub actions, I can run it also against the internal pipeline for the other databases, please ping me for that.

I will, for sure.

Regarding the SPIs: there is a small window for Keycloak 19 and possibly 20 were a change could be done a little bit easier with regards to backward compatibility, still the earlier the better.

Ok, will do this asap.

That being said, please ping me (@ahus1) on any PR you create to this. I'd also like to see a PR for the https://github.com/keycloak/keycloak-benchmark/tree/main/benchmark/src/main/scala/keycloak/scenario/admin with a scenario that could be similar to "CreateClients".

Will have a look at creating the Gatling scenario.

Thx!

Hi @davoustp ,Any updates on the realm creation PR and will it be part of next keycloak release ?.

Hi @sreekesh93,
I started to split all the changes into individual PRs, and some more to add benchmark scenario to https://github.com/keycloak/keycloak-benchmark/. The first PR showed up with a problem which I'm currently investigating.

Hi @davoustp
I built keycloak from you fork and deployed it with postgres
In my case perfomance for keycloak realm creation and admin console login doesn't look better
Maybe there are some point how to tune it?
I kept 3 pods of keycloak and uploaded up to 400 Realms and after I wasn't able to login admin console

This is Dockerfile I used
https://github.com/vasyan1992/Keycloak-from-source/blob/main/Dockerfile

Hi @vasyan1992 , could you give a try with the built-in Docker image from the forked repo instead of your own, by following the official build instructions?

This boils down to running the command below from the quarkus directory:

mvn -f ../pom.xml clean install -DskipTestsuite -DskipExamples -DskipTests \
&& cd container/ \
&& cp ../dist/target/keycloak-18.0.0-SNAPSHOT.tar.gz . \
&& docker build --build-arg KEYCLOAK_DIST=keycloak-18.0.0-SNAPSHOT.tar.gz . -t quay.io/keycloak/keycloak:18.0.0-SNAPSHOT \
&& cd ..

then execute the Docker command described at top of this document...
Let me know how this goes.

@vasyan1992
How is this going?
I received another notification from another user who had the same issue, so I'm wondering if there is something wrong into the instructions.
So I ran through the instructions again, and ran the script that creates 1k realms: I still get the expected performance gain.

The only thing that I could think of is that you're running the wrong image: the fork builds the quay.io/keycloak/keycloak:18.0.0-SNAPSHOT image locally, so that's the one to use of course, not the quay.io/keycloak/keycloak:17.0.0 image as described above (that was to perform the baseline measurement against the standard Keycloak distrib).

@davoustp that notifcation was maybe from me, I first tried to build and didn't see the performance improvements.
I realized I actually wasn't on the right branch KEYCLOAK-4593-investigations, that's why I didn't see the improvements.
I realized and deleted my comment here. On the right branch, it works!
Thanks for your work on this! <3

Up! I think this is still an issue

Hi @stianst - I made an attempt to apply some of the ideas to the map store in #13146. After a discussion with @hmlnarik and @vramik we agree that this is possible, but put this on hold until the tree store is available and the team capacity allows for it.

@stianst +1 to review if we can improve admin console (WhoAmI etc) and ideally fetch just the roles for the current realm. I suggested something similar above (point 6) some time ago in the comment #11074 (comment)

Hi there,
Even if optimizing the UI and related API endpoints to restrict role lookup to a single realm is desirable (in any case) and would help the UI-end of the stick, you'd still have the other side of it to cope with: the same role composite loading performance problem happens when creating realms using UI or API (and importing realms as well, btw, since this is the same underlying code).
So the scalability issue would still remain...
Cheers

@davoustp When creating realms, you currently need to have create-realms role of the master realm. So if we simplify in a manner that for example:

• Realm master have only single role admin . Guy in that role has permission to manage anything in any realm and also permission to create/delete realms. So like "admin of the world" . There won't be any other roles in the master realm and not thousands of composite roles like now
• When someone want fine-grained permission to administer individual realms, you need to have account in the realm itself.

In other words, no possibility for fine-grain permissions from the administrators of master realm. AFAIK that would work as the master realm is the only realm, which is causing troubles. The other individual realms have just a few roles to manage just the realm itself (roles of the realm-admin client of individual realms).

@mposolda That's not compatible with the scenario we are using Keycloak in. We provide Keycloak instances to our internal customers. We give our customers rights to manage their own realms from the master realm. At the same time, they cannot change anything in the master realm. This way, they cannot shoot themselves in the foot by making configuration changes in their realms - they will always have their guaranteed realm admin users we give them. At the same time, when troubleshooting issues, we use read-only users in the master realm with access to the customer realms, so we don't accidentally modify data. So the ability to have per-realm manageability from the master realm is really important for us.

@davoustp @stianst @mposolda @pedroigor @hmlnarik
Sorry for the multi-mention, but I'm desperately waiting for an answer on my original question:

Are there any plans to bring the improvements to production soon? We would need the scalibility improvements soon, since our customer-base is starting to grow steadily in the last months.

Hi @matt-rauch ,
As to your question, I'm still monitoring this scalability issue, but no progress so far that I'm aware of.
As far as I understood, some of the optimization ideas from this initiative may be brought to the new store which is also bringing additional goodies (such as zero-downtime upgrades), but I've got no visibility onto this - am kind of blind to what's happening here ;-)
I suspect running the benchmark scenario introduced with keycloak/keycloak-benchmark#60 against a KC with the new store would help check how far it gets...

@kkcmadhu Your workaround with the front load balancer sounds interesting and we consider to implement it ourself.

Could you elaborate how you configured the routing at the front load balancer to make multiple keycloak clusters look like one?
Are there any other restrictions to that approach?
What is the maximum number of keycloak realms you host per cluster?

I support about 8k realms, i pack about 300 to 500 realms per keycloak and run them behind envoy proxy.
every keycloak cluster ( we call it a farm) has 2 nodes, and becomes a envoy cluster endpoint..
we use envoy java control plane to dynamically add routes etc.. Been in prod for about 3 years and servers us well..

Some major challenge are

• there are more than one master realm now, and to create a new realm you wll have to dispatch the request to specific keycloak (known upfront), or get to specific keyclok and create realm there.
• once new realm is created one needs to configure your envoy and add route for the newly provisioned realm /like wise remove it after deprovision.
• the static identifier for each keycloak defers( after keycloak 17 i guess).. so you may need spl routing rules for static resources.
• You are now managing multiple keycloak, version upgrades could be trick... we choose to upgrade all our keycloak farms all at once. ( else you might end up maning different versions of keycloak , all under single url at once, and api compatibility issues could crop up).

Certainly cumbersome to begin with , but once you automate things its easy sail..
On latency, i did not see any big issues because of the extra envoy hop..

@kkcmadhu

Thanks for sharing the details of your setup we've actually done something similar in our design with multiple keycloaks routing the requests to the correct keycloak. However we've not been able to achieve 300-500 realms per keycloak we are seeing "slowness" with loading keycloak UI and authentication at around 150 realms. I'm starting to think that other factors weigh on this like the number of users/clients/roles per realm may weigh in on the slowness. I was curious if you don't mind sharing:

1. What CPU / Mem per node are you using? We are currently using 2 cpu x 4 GB of ram per node. we tried increase this but didn't notice any reasonable improvement in performance when facing the slowness
2. When you login to the admin console and the browser is loading the initial page with all the realms how long does this take on your larger keycloaks? We are seeing 5-10 second load times which is manageable but again we have a fraction of the realms you do. If we go with more realms that time gets slow and even the standard authentication times get longer.
3. How do you know when you've reached the limit of realms per cluster? Do you have a calculation or do you just judge it on slowness?

We are in the process of upgrading to Keycloak 21 and are hoping there might be some improvement we are currently on Keycloak 16.

Thanks for your time,
John

I have variety of cpu/mem based on customer segments, we use k8s, and our nodes are large as we have many other services running along with keycloak, typically we allocate a max of 2 core and 8 gig for large deployments and 1 core , 4 gig for others. At times we have upto 3 pods in a kecloak cluster (generally 2 for HA). we disable stickiness.
For static resources, i pool all of them into single envoy cluster and let all the static resource request go to any of the keycloak deployments ( its not much helpful but comes handy at times when load is high).
If you can leverage cloudflare/ cloud front etc those to help in static page access a bit(but again not very useful) in my opinion.

Our typical realm may not have more than 500 users, 1 or two identiy provider, and not more than 15 clients .. so never ran into the issues which you mentioned.

We restricts/ avoid using they keycloak console as much as possible and try to do stuff using api (in our case) we disable keycloak console access to customers too only our internal ops team uses keycloak admin console (that too very restricted). typical things like chaning url of client etc we do it using apis.

My observation is after keycloak 17+ there is significant degradation in realm creation time, dead locks (when realm creation and deletion ) is in progress at the same time.
In fact we saw the time to get token also increase, but thats not very high..

Typically the realm creation time doubles for every 100 realms in my opinion.. and there are about 50 ms extra for each each token requests , when you add about 50 realms or so..

In my opinion, if you avoid using keycloak admin console, and have some kind of retry mechanisms for your realm creation deletion etc. you will be able to handle a part of problem. But its certainly significant effort "outside" keycloak and takes a little time to build and mature.. and needs constant maintenance as and when keycloak are upgraded.

Answering your specific questions :

What CPU / Mem per node are you using? We are currently using 2 cpu x 4 GB of ram per node. we tried increase this but didn't notice any reasonable improvement in performance when facing the slowness

we use much bigger box 4 cpu, 8 or 16 gig k8s node, but its not dedicated to keycloak alone, we run variety of serivce there.

When you login to the admin console and the browser is loading the initial page with all the realms how long does this take on your larger keycloaks? We are seeing 5-10 second load times which is manageable but again we have a fraction of the realms you do. If we go with more realms that time gets slow and even the standard authentication times get longer.

Admin console, we avoid as much as possible and rely on the apis, We do not use cli much as you will have to log on the nodes, but in my opinion using the api are much better trade off (cli needs pod access, webconsole has known issues).

How do you know when you've reached the limit of realms per cluster? Do you have a calculation or do you just judge it on slowness?
From our past perf test, we know realm creation time doubles every 100 realms, We want realm creation to be in less than 5 mins, Realm deletion/is a batch process for us, our SLA is keycloak uptime, and decent response time for token/login calls, we found that the magic number is any where around 300 to 500 realms per keycloak for decent performance of login, since we have envoy in front, all our routes are persisted to a db, whenever a new route is added , we start throwing alerts when keycloak is 80% full ( say 400 realms), and we provision a new one, or aggressively cleanup "

I wouldn't say its super effective, but keeps us going, without burning midnight oils.

Thank you for the detailed comments. We do use the API for automation like creating deleting realms, clients, users.. The Admin Console is mostly for troubleshooting and manual effort. Thank you for the details on the keycloak 17+ upgrade slowness we will keep an eye out.

For the second point, there already exists the PR #8851

It is probable not compatible to the new admin console 🤔

Cool, as a first little thing then let's try to get that reviewed/merged :)

If anyone here wants to take a look at updating #8851 that'd be more than welcome :)

how is it different from whats already being offered by phaseII as keycloak extension?
https://github.com/p2-inc/keycloak-orgs

I would assume realm creation and deletion to happen quite rarely. I am curious what is your use case where this is happening often?

Don't want to answer for @kkcmadhu. But like to mention what I have observed: That realm creation/deletion issue is a problem when running automated tests in parallel (against the same keycloak instance). The main problem is deleting test realms when other - even unrelated tests on other realms - are still running.

Yes ,we too see it in automated test cases.
Besides, in I run a SaaS application and for isolation i allocate 1 realm per customers, when their trial periods expire we cleanup realms,
this is at time done in batch too. and the same time there are customers who are trying to signup (and hence create realms).

Agreed its not as frequent as login/token calls , but not improbable and in reality we have seen it quite a lot.

Thanks, that makes sense.

While org is a nice concept and helps in few use cases, it will not help SaaS applications which are multi tenant in nature.
To provide maximum isolation , We actually onboard each of our customer as a separate realm and let them manage their users, use-role association, Identity providers alone
All our SaaS offerings are actually clients in customers realm..
Say if we have 10 offering those become 10 clients. and a customer can do SSO across our product lines..

So org would not benefit use case where tenant isolations are priority like ours.

I fully agree to @kkcmadhu 's statement. It would not help us with our B2B SaaS service aswell, because our clients need the realm level isolation. In markets like Germany it is actually a hard requirment by the BSI (Bundesamt für Sicherheit in der Informationstechnik to provide this kind of isolation, when you service governmental clients.

For us, both solutions (organization/multiple realms) would be applicable.

org support exists in keycloak extension https://github.com/p2-inc/keycloak-orgs

To chime in on this discussion. We also use Keycloak in a B2B SaaS setup and create a separate realm per customer. The application runs on a different sub domain for each customer. So that would make our client setup interesting, because they would have to support either a long list of redirect URIs (not sure if I want a wild card here) or we would have to come up with a different naming pattern to ensure clients can live next to each other. At the moment they use the same alias in different realms.

Furthermore, we have differences in authentication flows and the login screen depending on the customer needs. Authentication flows could potentially be reused for different pattern, but would separate login screens work?

To me it sounds like either the organization concept is not sufficient for some of our use cases or it becomes another realm separation just with a different name. Then I wonder if the time is not better invested in improving the existing realm separation.

in my opinion, org is fairly stright forward to acheive (even today) with keycloak connect (identity broker) and few broker/identity mapper and few claims mapper while printing jwt token.. which is what we do to realize org structure presently.

I kind of get @stianst point of view, i think he is trying to emphasis master realm is already some kind of org structure where one can go ahead and create realms from master realm and master relam has clients representing each of the actual realm (xxx-realm-management) .. (and possibly most performance problems originates there)

But this master realm/ super admin relam is spl in nature, i dont think bring a full fledged "org" capability is a replacement of all realms and resource with in realms being managed from master realm.

Also.. With org, one need to consider about the token size.. even today, the master realm token tends to get really huge if you have lots of realm in your key cloak deployment.. The same could actually happen with org too..

I agree with other that "org" is totally different from complete realm isolation (and many of us will still need/heavily depdend on realm level isolations which keycloak presently offers)...

I get the point that "org" could make things easy from an operations point of view, however I am not really able to understand how "org" could actually solve the inherent performance issue with respect to large number of realms, and increase in token size and the dead lock issues and high latency in realm creation/deletion when a keycloak instance/cluster already has more than 50 or 100 realms in it.

From my PoV, we need and can improve multi-realm support to at least something better than today. Several attempts were made in the past but we never moved forward. There are some low-hanging fruits we can get from these past initiatives as they attack well-known issues when you have a multi-realm deployment.

IMO, without physical isolation the scalability of deployments will always be a problem, and depending on their realms workload you will end up with too many gears assigned to deployment so that CPU, RAM, Network, Database, and so forth are going to be pushed to their limits. For instance, in an HA deployment with an Active/Passive(Active) that could be even worse to maintain due to the amount of data going between the different sites, etc.

My point is, that scalability should be relative and constrained regardless. However, we definitely have improvements to today's runtime when doing multi-realm. These can perhaps be enough to solve most of the use cases discussed.

Yes, one of the key goals for 2024 is multi-tenancy. If we are going to focus on the org concept or support multiple realms, it is still unclear. More likely we will start with org due to some hard requirements we have in our backlog.

any ETAs we can look forward to? we are getting to the point of evaluating alternatives to keycloak because of this but the migration effort would very high and we would rather avoid it if this issue will be solved soon.

@pedroigor and @stianst , both of you seem to mention an "org concept" but to me it's kinda vague what this concept looks like? Is there a page or something that describes what it is compared to multi-realm? Perhaps we can also assist with some PRs to help push this forward if we'd only know what it is and where this org concept is tracked (if anybody is already working on it at all)