Users cache clears after creating client scope. #31045
Description
Before reporting an issue
- I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.
Area
infinispan
Describe the bug
While updating Keycloak to a distribution based on Quarkus, we encountered the following issue: in a clustered installation consisting of, for example, three nodes, creating a client scope on one node completely clears the user cache for the realm where the client scope was created on the other nodes.
This behavior has been observed in all versions after commit 7855b933909127937672c5ce60ede1e065b6bb2d
Version
19.0.0 or higher
Regression
- The issue is a regression
Expected behavior
The user cache content should not be cleared after creating a client scope.
Actual behavior
When creating a client scope on one node of the cluster, the user cache content for the realm where the client scope was created is completely cleared on the other nodes.
How to Reproduce?
- Deploy a Keycloak cluster of any version newer than 18.0.2 with user cache metrics enabled.
keycloak.conf
# cluster
cache=ispn
cache-config-file=cache-ispn.xml
# Observability
metrics-enabled=true
health-enabled=true
cache-ispn.xml (default, just enable statistics)
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:13.0 http://www.infinispan.org/schemas/infinispan-config-13.0.xsd"
xmlns="urn:infinispan:config:13.0">
<jgroups>
<stack name="tcpping" extends="tcp">
<TCP bind_port="7600" />
<TCPPING
initial_hosts="keycloak1[7600],keycloak2[7600],keycloak3[7600]"
port_range="0"
stack.combine="REPLACE"
stack.position="MPING"
/>
</stack>
</jgroups>
<cache-container name="keycloak" statistics="true">
<transport lock-timeout="60000" stack="tcpping"/>
<local-cache name="realms" simple-cache="true">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<local-cache name="users" simple-cache="true" statistics="true">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
...
- Create a test realm and several users in it.
- On all nodes of the cluster, perform a user search in the test realm through the admin web interface to populate the user cache.
- Retrieve the metric from each cluster node showing the number of entries in the user cache.
curl -s http://127.0.0.1:8082/auth/metrics | grep entries | grep users
vendor_cache_manager_keycloak_cache_users_statistics_approximate_entries{node="40f4b2ced666-5496"} 17.0
vendor_cache_manager_keycloak_cache_users_statistics_approximate_entries_in_memory{node="40f4b2ced666-5496"} 17.0
vendor_cache_manager_keycloak_cache_users_statistics_approximate_entries_unique{node="40f4b2ced666-5496"} 17.0
vendor_cache_manager_keycloak_cache_users_statistics_current_number_of_entries_in_memory{node="40f4b2ced666-5496"} 17.0
vendor_cache_manager_keycloak_cache_users_statistics_number_of_entries{node="40f4b2ced666-5496"} 17.0
- On one of the nodes, create a client scope in the test realm.
- Retrieve the metric from each cluster node again, showing the number of entries in the user cache. As a result, we will see that on the nodes where the client scope was not created, the entries in the user cache have been cleared.
Anything else?
No response