OLM v0.31.0 (packageserver CSV): "patch" and "watch" verbs ATM missing for "apiservices" from "apiregistration.k8s.io" API group as necessary "clusterPermissions" in packageserver CSV #3538
Description
Bug Report
What did you do?
Given the standard OLM ClusterRole definition is pretty wide (meaning the asterisk notation in resources: and verbs: fields ATM), we are using list of the RBAC rules from the packageserver's spec.install.spec.clusterPermissions CSV section:
- https://github.com/operator-framework/operator-lifecycle-manager/blob/master/deploy/chart/templates/0000_50_olm_01-olm-operator.serviceaccount.yaml#L6
- https://github.com/operator-framework/operator-lifecycle-manager/blob/master/deploy/upstream/manifests/0.18.3/0000_50_olm_15-packageserver.clusterserviceversion.yaml#L37
to enumerate (minimize) the list of K8s resources & verbs, that are required to install packageserver correctly.
This works (seem to work) pretty well, except the packageserver to stay at Failed state, instead of correctly getting to the Succeeded state.
Our testing has shown, besides the RBAC rules, currently listed in clusterPermissions section of packageserver CSV, the following RBAC rule is required for packageserver CSV yet, in order to get it installed correctly:
- apiGroups:
- "apiregistration.k8s.io"
resources:
- "apiservices"
verbs:
- patch
- watch
Without this rule being present it's not possible to get packageserver to the Succeeded state (using solely the RBAC rules specified in clusterPermissions section of the packageserver CSV).
Therefore wondering, if the aforementioned rule should be added as a necessary requirement (to the packageserver's spec.install.spec.clusterPermissions CSV section) to get packageserver installed correctly OOTB?
What did you expect to see?
packageserver CSV reached the Succeeded state & packageserver pods are running correctly:
# kubectl -n <namespace> get csv
NAME DISPLAY VERSION REPLACES PHASE
packageserver Package Server <chart.version> Succeeded
What did you see instead? Under which circumstances?
packageserver CSV "stays" in Failed state:
kubectl -n <namespace> get csv
NAME DISPLAY VERSION REPLACES PHASE
packageserver Package Server <chart.version> Failed
With the following details / output being available, describing the actual packageserver CSV
Phase: InstallReady
Reason: ComponentUnhealthy
Last Transition Time: 2025-03-17T10:48:28Z
Last Update Time: 2025-03-17T10:48:28Z
Message: install strategy failed: install strategy failed: apiservices.apiregistration.k8s.io "v1.packages.operators.coreos.com" is forbidden: User "system:serviceaccount:<namespace>:olm-operator-serviceaccount" cannot patch resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope
Phase: Failed
Reason: InstallComponentFailedNoRetry
Last Transition Time: 2025-03-17T10:49:20Z
Last Update Time: 2025-03-17T10:49:20Z
Message: Cluster resources changed state
Phase: Pending
Reason: DetectedClusterChange
Last Transition Time: 2025-03-17T10:49:21Z
Last Update Time: 2025-03-17T10:49:21Z
Message: all requirements found, attempting install
Phase: InstallReady
Reason: AllRequirementsMet
Last Transition Time: 2025-03-17T10:49:21Z
Last Update Time: 2025-03-17T10:49:21Z
Message: install strategy failed: install strategy failed: apiservices.apiregistration.k8s.io "v1.packages.operators.coreos.com" is forbidden: User "system:serviceaccount:<namespace>:olm-operator-serviceaccount" cannot patch resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope
Phase: Failed
Reason: InstallComponentFailedNoRetry
Last Transition Time: 2025-03-17T10:49:21Z
Last Update Time: 2025-03-17T10:49:21Z
Message: install strategy failed: install strategy failed: apiservices.apiregistration.k8s.io "v1.packages.operators.coreos.com" is forbidden: User "system:serviceaccount:<namespace>:olm-operator-serviceaccount" cannot patch resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope
Phase: Failed
Reason: InstallComponentFailedNoRetry
Requirement Status:
Group: operators.coreos.com
Kind: ClusterServiceVersion
Message: CSV minKubeVersion (1.11.0) less than server version (v1.29.13)
Name: packageserver
Status: Present
Version: v1alpha1
Group: apiregistration.k8s.io
Kind: APIService
Message:
Name: v1.packages.operators.coreos.com
Status: DeploymentFound
Version: v1
Dependents:
Group: rbac.authorization.k8s.io
Kind: PolicyRule
Message: cluster rule:{"verbs":["create","get"],"apiGroups":["authorization.k8s.io"],"resources":["subjectaccessreviews"]}
Status: Satisfied
Version: v1
Group: rbac.authorization.k8s.io
Kind: PolicyRule
Message: cluster rule:{"verbs":["get","list","watch"],"apiGroups":[""],"resources":["configmaps"]}
Status: Satisfied
Version: v1
Group: rbac.authorization.k8s.io
Kind: PolicyRule
Message: cluster rule:{"verbs":["get","list","watch"],"apiGroups":["operators.coreos.com"],"resources":["catalogsources"]}
Status: Satisfied
Version: v1
Group: rbac.authorization.k8s.io
Kind: PolicyRule
Message: cluster rule:{"verbs":["get","list"],"apiGroups":["packages.operators.coreos.com"],"resources":["packagemanifests"]}
Status: Satisfied
Version: v1
Group:
Kind: ServiceAccount
Message:
Name: olm-operator-serviceaccount
Status: Present
Version: v1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
...
Normal AllRequirementsMet 2m45s operator-lifecycle-manager all requirements found, attempting install
Warning InstallComponentFailedNoRetry 2m44s operator-lifecycle-manager install strategy failed: install strategy failed: apiservices.apiregistration.k8s.io "v1.packages.operators.coreos.com" is forbidden: User "system:serviceaccount:<namespace>:olm-operator-serviceaccount" cannot patch resource "apiservices" in API group "apiregistration.k8s.io" at the cluster scope
Environment
- operator-lifecycle-manager version:
OLM v0.31.0 from upstream's release
-
Kubernetes version information:
Server Version: v1.29.13 -
Kubernetes cluster kind:
kubeadmbased installation of top of Ubuntu OS
Possible Solution
Add / list the following rule in spec.install.spec.clusterPermissions.rules section of the packageserver CSV:
- apiGroups:
- "apiregistration.k8s.io"
resources:
- "apiservices"
verbs:
- patch
- watch
to get the packageserver CSV to install / operate correctly (report reaching the Succeeded state properly)
in this scenario.
Clearly, there's a workaround to add this rule manually to the main OLM ClusterRole definition, when not wanting to use the upstream for (it's too wide and not actually needed for our purpose).
But on the other hand, the clusterPermissions section of packageserver CSV (namely spec.install.spec.clusterPermissions.rules of it) should list all the requirements, that should be met, to get the packageserver CSV to work properly (that's the actual point / meaning of the clusterPermissions section of the specific CSV)