Keycloak operator lacks RBAC for Pods #21814
Description
Before reporting an issue
- I have searched existing issues
- I have reproduced the issue with the latest nightly release
Area
operator
Describe the bug
Keycloak operator (version below) deployed on OpenShift v4.12 starts throwing errors complainig that its ServiceAccount keycloak-operator does not have access to list Pods. I am not sure what is the effective impact here, since Keycloak CR seems being properly handled (and Keycloak operand seems working fine).
Looking into the source repo, https://github.com/keycloak/keycloak/blob/main/operator/src/main/kubernetes/kubernetes.yml#L5-L57 it looks like indeed RBAC for Pods is not listed. Reading thru the code, seems that the impact is that Keycloak Operator cannot check deployed Pod's status and report any issues back?
__ ____ __ _____ ___ __ ____ ______
--/ __ \/ / / / _ | / _ \/ //_/ / / / __/
-/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
2023-07-19 08:38:50,677 INFO [io.qua.ope.run.OperatorProducer] (main) Quarkus Java Operator SDK extension 6.2.1 (commit: 2ba533d on branch: 2ba533dc2c2cf7ab3083a641f7a1badca5d68a62) built on Tue Jul 04 13:00:42 GMT 2023
2023-07-19 08:38:50,723 INFO [io.jav.ope.Operator] (main) Registered reconciler: 'keycloakcontroller' for resource: 'class org.keycloak.operator.crds.v2alpha1.deployment.Keycloak' for namespace(s): [keycloak-odlm-cert]
2023-07-19 08:38:50,732 INFO [io.jav.ope.Operator] (main) Registered reconciler: 'keycloakrealmimportcontroller' for resource: 'class org.keycloak.operator.crds.v2alpha1.realmimport.KeycloakRealmImport' for namespace(s): [keycloak-odlm-cert]
2023-07-19 08:38:50,732 INFO [io.qua.ope.run.AppEventListener] (main) Starting operator.
2023-07-19 08:38:50,732 INFO [io.jav.ope.Operator] (main) Operator SDK 4.4.0 (commit: 08f8d85) built on Fri Jun 23 16:25:52 GMT 2023 starting...
2023-07-19 08:38:50,732 INFO [io.jav.ope.Operator] (main) Client version: 6.7.2
2023-07-19 08:38:50,736 INFO [io.jav.ope.pro.Controller] (Controller Starter for: keycloakcontroller) Starting 'keycloakcontroller' controller for reconciler: org.keycloak.operator.controllers.KeycloakController, resource: org.keycloak.operator.crds.v2alpha1.deployment.Keycloak
2023-07-19 08:38:50,736 INFO [io.jav.ope.pro.Controller] (Controller Starter for: keycloakrealmimportcontroller) Starting 'keycloakrealmimportcontroller' controller for reconciler: org.keycloak.operator.controllers.KeycloakRealmImportController, resource: org.keycloak.operator.crds.v2alpha1.realmimport.KeycloakRealmImport
2023-07-19 08:38:50,775 WARN [io.fab.kub.cli.dsl.int.VersionUsageUtils] (InformerWrapper [keycloakrealmimports.k8s.keycloak.org/v2alpha1] 34) The client is using resource type 'keycloakrealmimports' with unstable version 'v2alpha1'
2023-07-19 08:38:50,777 WARN [io.fab.kub.cli.dsl.int.VersionUsageUtils] (InformerWrapper [keycloaks.k8s.keycloak.org/v2alpha1] 35) The client is using resource type 'keycloaks' with unstable version 'v2alpha1'
2023-07-19 08:38:51,728 INFO [io.jav.ope.pro.Controller] (Controller Starter for: keycloakrealmimportcontroller) 'keycloakrealmimportcontroller' controller started
2023-07-19 08:38:51,981 INFO [io.jav.ope.pro.Controller] (Controller Starter for: keycloakcontroller) 'keycloakcontroller' controller started
2023-07-19 08:38:51,992 INFO [org.key.ope.con.KeycloakController] (ReconcilerExecutor-keycloakcontroller-59) --- Reconciling Keycloak: example-keycloak in namespace: keycloak-odlm-cert
2023-07-19 08:38:52,039 INFO [org.key.ope.con.KeycloakController] (ReconcilerExecutor-keycloakcontroller-60) --- Reconciling Keycloak: example-keycloak in namespace: keycloak-odlm-cert
2023-07-19 08:38:52,049 INFO [io.quarkus] (main) keycloak-operator 22.0.1 on JVM (powered by Quarkus 3.2.0.Final) started in 3.303s. Listening on: http://0.0.0.0:8080
....
023-07-19 08:38:54,506 INFO [org.key.ope.con.KeycloakDeployment] (ReconcilerExecutor-keycloakcontroller-69) Found config secrets names: []
2023-07-19 08:38:54,536 INFO [org.key.ope.con.KeycloakDeployment] (ReconcilerExecutor-keycloakcontroller-69) Existing Deployment found, handling migration
2023-07-19 08:38:54,591 ERROR [org.key.ope.con.KeycloakController] (ReconcilerExecutor-keycloakcontroller-69) --- Error reconciling: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.30.0.1:443/api/v1/namespaces/keycloak-odlm-cert/pods?labelSelector=controller-revision-hash%3Dexample-keycloak-699fccdc56%2Capp%3Dkeycloak%2Capp.kubernetes.io%2Fmanaged-by%3Dkeycloak-operator%2Capp.kubernetes.io%2Finstance%3Dexample-keycloak. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:keycloak-odlm-cert:keycloak-operator" cannot list resource "pods" in API group "" in the namespace "keycloak-odlm-cert".
at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:518)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:420)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:388)
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:92)
at org.keycloak.operator.controllers.KeycloakDeployment.checkForPodErrors(KeycloakDeployment.java:371)
at org.keycloak.operator.controllers.KeycloakDeployment.updateStatus(KeycloakDeployment.java:350)
at org.keycloak.operator.controllers.KeycloakController.reconcile(KeycloakController.java:116)
at org.keycloak.operator.controllers.KeycloakController.reconcile(KeycloakController.java:49)
at org.keycloak.operator.controllers.KeycloakController_ClientProxy.reconcile(Unknown Source)
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:152)
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:110)
at io.javaoperatorsdk.operator.api.monitoring.Metrics.timeControllerExecution(Metrics.java:219)
at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:109)
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:140)
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:121)
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:91)
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:64)
at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:409)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.30.0.1:443/api/v1/namespaces/keycloak-odlm-cert/pods?labelSelector=controller-revision-hash%3Dexample-keycloak-699fccdc56%2Capp%3Dkeycloak%2Capp.kubernetes.io%2Fmanaged-by%3Dkeycloak-operator%2Capp.kubernetes.io%2Finstance%3Dexample-keycloak. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:keycloak-odlm-cert:keycloak-operator" cannot list resource "pods" in API group "" in the namespace "keycloak-odlm-cert".
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:671)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:651)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:597)
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:560)
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:140)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:52)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
at io.fabric8.kubernetes.client.vertx.VertxHttpRequest.lambda$null$1(VertxHttpRequest.java:122)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:246)
at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:76)
at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250)
at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:444)
at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:255)
at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:134)
at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:708)
at io.vertx.core.impl.EventLoopContext.lambda$execute$2(EventLoopContext.java:78)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 more
Version
22.0.1
Expected behavior
No errors observed in the logs
Actual behavior
Observed errors / stack traced in the logs
How to Reproduce?
Subscript to Keycloak operator, create sample Keycloak CR, wait for its reconcile and then eventually errors start popping up
Anything else?
No response