Is your enhancement related to a problem? Please describe.

We have all the upgrade logic implemented in controllers and embedded in the built harvester binary. This is good for a one-click experience but awkward for developers who maintain the product and want to deliver new features or bug fixes promptly. We found that we often need to wait for the subsequent two releases to get feedback on features and fixes relevant to the upgrade that will be delivered in the next release. For instance, #2777 is a great example. Changes in the upgrade pipeline before phase three will be subject to this limitation because the newly implemented logic only exists in the new version of Harvester, and the old one is in charge in phase one and phase two. This also causes headaches for testing efforts to validate the upgrade-related issue. As a workaround, we often need to do it in a "same-version upgrade" flavor.

Describe the solution you'd like

As a result, we are tempted to resolve this issue by introducing a stand-alone upgrade manager, which consists of the original controllers separated from the harvester binary and can be deployed separately. The original, embedded upgrade controllers will be deprecated and only contain a minimal set of duties, including ISO image preparation and upgrade manager deployment. These tasks are unlikely to change and have the least amount of configuration, so we don't bother fiddling with them.

Describe alternatives you've considered

N/A

Additional context

With this enhancement and the other one, #7101, the Harvester Upgrade will have three pieces collaborating to provide a fast yet easy-to-adapt user and dev experience.

1. upgrade-shim: A set of logic implemented alongside all harvester controllers, responsible for a relatively simple process that is unlikely to change frequently in an early stage of Harvester Upgrade
2. upgrade-repo: A highly available central hub for Harvester Upgrade components to get what they want, including the metadata and image archives
3. upgrade-manager: A collection of upgrade-relevant Kubernetes custom controllers, managing the Harvester Upgrade's lifecycle

Despite the upgrade-shim being bundled with other harvester controllers in the harvester binary, both upgrade-repo and upgrade-manager can be colocated in the same pod (different containers) to reduce the overhead.