-
Notifications
You must be signed in to change notification settings - Fork 7.7k
[KEYCLOAK-16600] - Improvements when loading many realms #7712
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
@pedroigor Thanks. I was thinking about using DefaultLazyLoader for improve around caching of objects related to realm model like realms, clients etc. So this look to me like the step in right direction. I've did some testing of the PR and did not spot any obvious issue. So looks good to me. |
|
@mposolda I'm still looking at some other places we might improve. I'll hold a bit more. |
891bd13 to
3af024b
Compare
3af024b to
c502514
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pedroigor Thank you, this PR looks very well and apart from few minor comments inline I'm happy to have it in. Good job!
model/infinispan/src/main/java/org/keycloak/models/cache/infinispan/LazyLoader.java
Outdated
Show resolved
Hide resolved
model/infinispan/src/main/java/org/keycloak/models/cache/infinispan/DefaultLazyLoader.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Avoid creation of new object.
| MultivaluedHashMap<String, AuthenticationExecutionModel> flows = new MultivaluedHashMap<>(); | |
| for (AuthenticationFlowModel flow : getAuthenticationFlowList(() -> r)) { | |
| flows.put(flow.getId(), new LinkedList<>()); | |
| r.getAuthenticationExecutionsStream(flow.getId()).forEachOrdered(execution -> { | |
| flows.add(flow.getId(), execution); | |
| }); | |
| } | |
| return flows; | |
| return new MultivaluedHashMap<>(getAuthenticationFlowList(() -> r).stream() | |
| .map(AuthenticationFlowModel::getId) | |
| .collect(Collectors.toMap( | |
| id -> id, | |
| id -> r.getAuthenticationExecutionsStream(id).collect(Collectors.toList())) | |
| )); |
Similarly for the following methods.
I wonder whether MultivaluedHashMap actually makes any sense anymore, and whether we can do just with Map<String, List<AuthenticationExecutionModel>>? That would simplify the code even more
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you. Yet I still wonder about the following:
I wonder whether MultivaluedHashMap actually makes any sense anymore, and whether we can do just with Map<String, List>? That would simplify the code even more
WDYT?
Also the other methods below this one should be changed similarly
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are other places we can improve in caches. We should probably have a separate task for it. I would leave it as is and handle this one and others later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ack, could you please introduce a JIRA for that?
...nfinispan/src/main/java/org/keycloak/models/cache/infinispan/entities/CachedClientScope.java
Outdated
Show resolved
Hide resolved
services/src/main/java/org/keycloak/services/resources/admin/AdminConsole.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This call produces an array of hashes like [{id: ..., realm: ...}, ...{id: ..., realm: ...}]. This contains lot of garbage and can be optimized to only return array of pairs like [["id1", "realm1"], ["id2", "realm2], ...]
Could there be an endpoint "/list" that the following implementation:
@GET
public Stream<String[]> listRealms() {
return session.realms().getRealmsStream()
map(r -> return new String[] {r.getId(), r.getName});
}This would remove all the logic linked to serialization of RealmRepresentation which would in turn lead to a more performant behaviour.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't that mean changing the console too? If so, I'm not sure if the effort justifies the performance gain.
I would leave it as is, please. Later we can see how to improve it even more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right, that could mean changing the console as well. Could the parameter have a more descriptive name, like idNameOnly?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Basic can mean more than id and name. Ideally, we should be following the rest guidelines so that we can have better control over the response. But that is another work. I would just leave as is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For something else than full object representation, there is already used briefRepresentation. Please keep parameter names consistent across code base
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with renaming the parameter to briefRepresentation we are already using it in many places. One more question, should we default to true? I am not sure about all places where this endpoint is used, but I can't imagine any usecase where it is necessary to obtain all realms in one place, so true as the default value makes sense to me. WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll change the name. A big one though :) But if we are using at other places, so let's keep consistency. Hope we can sort this out better when we review our APIs based on the guidelines.
In regards to default to true, that could break backward compatibility. I would leave it as is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mhajas, I believe this is correct but would like you to check
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good to me, I like it. However, we should probably add it also to AbstractStorageManager, it should be used everywhere in future, so I think this is the correct place to add it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mhajas I've added the check at that method. Is that what you are looking for?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pedroigor yes, exactly. Thanks!
model/infinispan/src/main/java/org/keycloak/models/cache/infinispan/entities/CachedClient.java
Outdated
Show resolved
Hide resolved
c502514 to
c67fc58
Compare
|
Thanks @pedroigor for the changes! Please see updates inline |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pedroigor Thank you very much for the PR. The code looks really good, it will certainly make work with realms much faster.
However, I am not sure if it happens only on my local machine, but I am experiencing some issues when using the admin console. Could you please check following steps:
- I added one comment in code which is probably causing one of issues.
- When I create a new realm as master admin user, I see only a weird blank realm management page (similar as one I added to the comment in code) and also I can't see the created realm in the list of realms unless I reload the page. After the reload the admin console seems to be back to normal.
I would like to suggest running ui tests on this PR as it is changing admin console.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pedroigor Here we are skipping the authorization. It seems it is causing that when you open admin console as an administrator of some realm (other than master) you are able to see that there are other realms.
You can reproduce as follows:
- Create realm1
- Create realm2
- Create admin user in realm2 and assign him with all realm-management roles
- Log in to admin console of realm2 (http://localhost:8081/auth/admin/realm2/console) as user you created
- Previously you could see only realm2, now you can see all realms and you can choose which one you want to manage (all except realm2 will show forbidden)
- When I click realm2 I am seeing only this (I am not sure this is caused by this method though):
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mhajas Thanks for all the details about the issue. Looks like I missed the checks for filtering realms. It should be fixed now.
I'm not sure why you did not get the menu though. You were supposed to get that only after creating a new realm. Which now is fixed.
I also tried to make some improvements when accessing a particular realm when using admins other than master admin.
I agree we should run UI tests. Please let me know if you spot anything else because I know our UI tests do not provide full coverage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mhajas Btw, if you are testing performance as well, you should also consider increasing the number of entries in both realms and users cache.
My latest changes should reduce the time taken to get a token to ms, instead of seconds.
I can share with you the database dump I'm using for tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @pedroigor. It seems all issues I mentioned are fixed now. I played with it a little more and there is one more change in behaviour. When you start fresh keycloak with an admin user, after you navigate to admin console and log in, admin console is always asking what realm you want to manage (at #/realms), even when there is only one, while previously it opened #/realms/master. Is this change intentional?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mhajas It should be fixed now.
c43bd48 to
3ba77fa
Compare
900b3bb to
a1ee184
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for all the changes @pedroigor! Lets wait for pipeline runs to finish and if there are no additional failures this is ready for merge from my point of view.
be0e45a to
b5315ed
Compare
|
Also fixing some console tests. |
0e39f6c to
ac8ae81
Compare
|
I'm also including here https://issues.redhat.com/browse/KEYCLOAK-16825. It improves a lot the time taken to calculate admin permissions. |
c9df644 to
171475e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me now, thanks @pedroigor! So far only missing pipeline results prevent me from approving this one.
Can you please share the links to the pipeline run.
If there are any additional optimizations, could those be made in a fresh PR so that this one can be merged soon once all other reviewers agree it's mergeable? It has impact on the realm map storage work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice stuff, @pedroigor! :)
Maybe it's too late for a review (sorry for that!) but I added a few comments.
Do we have any perf tests results?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch. I'll fix it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just out of curiosity, why aren't the scopes lazily loaded (and lazily cached)? Is it better for performance (scopes are always needed)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Due to some issues discussed in this PR. Mainly, the possibility to have inconsistencies due to how we use it in predicates.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't we rather throw 404?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We fallback to the realm inferred from the path if none/wrong was provided.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still feel that from the REST API perspective it would make more sense to explicitly tell that the realm doesn't exist rather than return something the user didn't ask for.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But I don't want to block it, so let's leave it like it is. :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't we rather throw 404?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In case the realm was inferred from the path.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| return Response.ok(new WhoAmI(user.getId(), realm.getName(), displayName, createRealm, roles, locale)).build(); | |
| return Response.ok(new WhoAmI(user.getId(), currentRealm.getName(), displayName, createRealm, roles, locale)).build(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a bit nit picking here. I wonder, could we use a different way to supply the cache objects with models – something else than a method attribute? IMHO this unnecessarily makes the code less clean and harder to read. Maybe somehow use DI instead? I don't know... maybe it's a stupid idea.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if makes sense. At least not now considering the scope.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the reason to remove this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With changes to admin console code, there is no 403 like that. I'm not sure why it was there. Look at the final assertion, that is what makes sense to assert.
Yes, please try loading 1k+ realms and open admin console. Also, compare time to migrate. In terms of runtime, this is a WIP due to some investigation happening on top of the master codebase. |
171475e to
86d2a89
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@pedroigor Thanks for the update. Changes look good to me, we just need to make sure test pipelines passed (if we haven't already).
|
Hey folks, thanks for this PR! Any updates about the remaining work to do? 😬 |
|
@hadrien-toma I would also like to re-take this one as it certainly improves multi-realm support. But the plans changed a bit and some of the improvements here should be part of the new storage. If we decide for it I would need to review it again (although the original change set already got a positive feedback) and updated it accordingly. |
Thanks @pedroigor, in fact multi-realms are highly wanted haha. Do you have any clues about the road map on this subject? |
|
@hadrien-toma Thank you. @hmlnarik Can give you more details about the roadmap. |
Running tests using a database populated with the default settings from the
keycloak-benchmarktool. Using 1K realms.whoamiendpoint, but for a single and the current realm being managed from the admin consoleExpected results:
Next steps: