feat(upgrade): pausable node upgrade #924

starbops · 2025-11-20T09:34:30Z

Introduce the node upgrade pause feature by describing how to pause and unpause node upgrades.

Problem:

The new node upgrade pause feature lacks documentation.

Solution:

Outline the new settings and provide steps to pause/unpause node upgrades.

Related Issue(s):

harvester/harvester#8980

Test plan:

Additional documentation or context

github-actions · 2025-11-20T09:58:23Z

Name	Link
🔨 Latest commit	`448270a`
😎 Deploy Preview	https://694371b2536c1152c49a0ed3--harvester-preview.netlify.app

Introduce the node upgrade pause feature by describing how to pause and unpause node upgrades. Signed-off-by: Zespre Chang <[email protected]>

w13915984028

LGTM, thanks.

w13915984028 · 2025-11-26T09:05:56Z

docs/upgrade/automatic.md

+
+# Update the annotation to unpause the node
+$ kubectl -n harvester-system annotate --overwrite upgrades hvst-upgrade-6mcwv harvesterhci.io/node-upgrade-pause-map='{"charlie-1-tink-system":"unpause","charlie-2-tink-system":"pause","charlie-3-tink-system":"pause"}'
+upgrade.harvesterhci.io/hvst-upgrade-6mcwv annotate


L310 is better to be removed

w13915984028 · 2025-11-26T09:12:35Z

docs/upgrade/automatic.md

+
+:::info
+
+Harvester applies the pause-relevant configuration during upgrade initialization. After that, any further changes to the fields under the `nodeUpgradeOption` option will not affect the current upgrade and can only take effect upon the next upgrade.


add to here or other places, mention that when the upgrade is ongoing, as long as a node is not upgraded yet, it can be freely added or removed to/from the upgrade object's annotation; the upgrade makes decision upon the upgrade object

I suppose this can be considered an advanced tip for now, since we don't have a counterpart UI component for it. The main entrance for the pause node upgrade feature is through configuring the upgrade-config setting.

ihcsim

content lgtm - thanks

ihcsim · 2025-11-26T19:24:20Z

docs/advanced/settings.md


+- `nodeUpgradeOption`: Options for the node upgrading phase.
+
+  In the node upgrade phase, Harvester upgrades each node's RKE2 and operating system autonomously. Harvester tries to migrate all virtual machines running on the to-be-upgraded node to other nodes. Harvester will shut down any virtual machine considered non-live-migratable to ensure the node upgrade goes smoothly. The `nodeUpgradeOption` provides customizability for how Harvester should behave upon node upgrades.


autonomously

I think if we are trying to convey that the operation either completes fully or fails, then "atomic" might be closer than "autonomous". So maybe something like:

"The node upgrade is an atomic operation, which includes upgrading the node's RKE2 components and operating system. The upgrade is either fully completed or failed, with no half-finished state."

ihcsim · 2025-11-26T19:33:16Z

docs/upgrade/automatic.md

+hvst-upgrade-6mcwv   4h16
+
+# Update the annotation to unpause the node
+$ kubectl -n harvester-system annotate --overwrite upgrades hvst-upgrade-6mcwv harvesterhci.io/node-upgrade-pause-map='{"charlie-1-tink-system":"unpause","charlie-2-tink-system":"pause","charlie-3-tink-system":"pause"}'


To confirm, removing charlie-1-tink-system works too, right?

No. If you do so, the upgrade controller does not know what node has been removed from the annotation. As a result, the relevant machine-plan secret cannot be enqueued for reconciliation. The upgrade holds still until you "touch" the corresponding machine-plan secret.

jillian-maroket

Review done

docs/upgrade/automatic.md

docs/advanced/settings.md

Co-authored-by: Jillian Maroket <[email protected]> Signed-off-by: Zespre Chang <[email protected]>

Copilot

Pull request overview

This PR introduces documentation for the new node upgrade pause feature available in Harvester v1.7.0. The feature allows administrators to pause automatic node upgrades for specific nodes or all nodes in the cluster, enabling manual maintenance or verification tasks to be performed before resuming the upgrade process.

Key changes:

Added comprehensive documentation for the nodeUpgradeOption setting configuration
Provided step-by-step instructions for pausing and resuming node upgrades
Included kubectl command examples and UI screenshots for visual confirmation

Reviewed changes

Copilot reviewed 2 out of 4 changed files in this pull request and generated 6 comments.

File	Description
docs/upgrade/automatic.md	Added new "Customize Node Upgrade" section with detailed instructions on pausing/resuming node upgrades, including kubectl examples and UI screenshots
docs/advanced/settings.md	Updated `upgrade-config` setting documentation to include the new `nodeUpgradeOption` field with its sub-fields (`mode` and `pauseNodes`), and reformatted the default value as JSON for better readability

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-17T03:26:06Z

docs/upgrade/automatic.md

+You can use the `nodeUpgradeOption` option in the [`upgrade-config`](advanced/settings.md#upgrade-config) setting to pause node upgrades.
+
+- Pause for all nodes in the cluster: Change the value of the `mode` field to `manual`.
+- Pause for specific nodes: List the node names in the `pauseNodes` field. Nodes not included in the list are automatically upgraded.


The documentation is incomplete regarding the relationship between mode and pauseNodes. Line 265 states you can pause specific nodes by listing them in pauseNodes, but according to settings.md line 858, this only works when mode is set to manual. When mode is auto, the pauseNodes field is ignored. This important prerequisite should be mentioned here to avoid confusion.

Suggested change

- Pause for specific nodes: List the node names in the `pauseNodes` field. Nodes not included in the list are automatically upgraded.

- Pause for specific nodes: When `mode` is set to `manual`, list the node names in the `pauseNodes` field. Nodes not included in the list are automatically upgraded. When `mode` is `auto`, the `pauseNodes` field is ignored.

docs/upgrade/automatic.md

Copilot · 2025-12-17T03:26:07Z

docs/upgrade/automatic.md

+
+### Pausing Node Upgrades
+
+You can use the `nodeUpgradeOption` option in the [`upgrade-config`](advanced/settings.md#upgrade-config) setting to pause node upgrades.


The link path is incorrect. Since automatic.md is in the docs/upgrade directory and settings.md is in docs/advanced, the relative path should start with "../" to navigate up one directory level. The correct path should be "../advanced/settings.md#upgrade-config" (consistent with other references to settings.md in this file on lines 92 and 108).

Suggested change

You can use the `nodeUpgradeOption` option in the [`upgrade-config`](advanced/settings.md#upgrade-config) setting to pause node upgrades.

You can use the `nodeUpgradeOption` option in the [`upgrade-config`](../advanced/settings.md#upgrade-config) setting to pause node upgrades.

docs/advanced/settings.md

Copilot · 2025-12-17T03:26:08Z

docs/advanced/settings.md

+
+    :::info important
+
+    Upgrading of nodes listed in this field is _paused definitely_ until you take specific actions to [resume the process](upgrade/automatic.md#how-to-resume-a-node-to-continue-with-node-upgrade). Given that Harvester upgrades nodes sequentially, this implies that the entire upgrade progress is paused as well. 


The phrase "paused definitely" appears to be a spelling error. The intended word is likely "paused indefinitely" (meaning paused for an unlimited time until action is taken), not "definitely" (meaning certainly).

Suggested change

Upgrading of nodes listed in this field is _paused definitely_ until you take specific actions to [resume the process](upgrade/automatic.md#how-to-resume-a-node-to-continue-with-node-upgrade). Given that Harvester upgrades nodes sequentially, this implies that the entire upgrade progress is paused as well.

Upgrading of nodes listed in this field is _paused indefinitely_ until you take specific actions to [resume the process](upgrade/automatic.md#how-to-resume-a-node-to-continue-with-node-upgrade). Given that Harvester upgrades nodes sequentially, this implies that the entire upgrade progress is paused as well.

Signed-off-by: Zespre Chang <[email protected]>

docs/advanced/settings.md

Signed-off-by: Zespre Chang <[email protected]>

brandboat

LGTM, thanks!

starbops mentioned this pull request Nov 20, 2025

[FEATURE] Pause-ability of node upgrades harvester/harvester#8980

Closed

feat(upgrade): pausable node upgrade

bedeb9c

Introduce the node upgrade pause feature by describing how to pause and unpause node upgrades. Signed-off-by: Zespre Chang <[email protected]>

starbops force-pushed the feat-8980 branch from cab4a8c to bedeb9c Compare November 20, 2025 16:04

starbops marked this pull request as ready for review November 20, 2025 16:04

github-actions bot requested review from akashraj4261, dariavladykina and jillian-maroket November 20, 2025 16:04

github-actions bot assigned starbops Nov 20, 2025

starbops requested review from akashraj4261, brandboat, dariavladykina, ihcsim and w13915984028 and removed request for akashraj4261 and dariavladykina November 20, 2025 16:04

w13915984028 previously approved these changes Nov 26, 2025

View reviewed changes

jillian-maroket removed request for akashraj4261 and dariavladykina November 26, 2025 10:02

ihcsim previously approved these changes Nov 26, 2025

View reviewed changes

jillian-maroket reviewed Nov 27, 2025

View reviewed changes

Apply suggestions from code review

d08d0db

Co-authored-by: Jillian Maroket <[email protected]> Signed-off-by: Zespre Chang <[email protected]>

starbops dismissed stale reviews from ihcsim and w13915984028 via d08d0db December 17, 2025 03:21

Copilot AI review requested due to automatic review settings December 17, 2025 03:21

Copilot started reviewing on behalf of starbops December 17, 2025 03:21 View session

Copilot AI reviewed Dec 17, 2025

View reviewed changes

starbops added 2 commits December 17, 2025 11:36

feat(upgrade): add a tip for pause node upgrade function

6fb6116

Signed-off-by: Zespre Chang <[email protected]>

fix(upgrade): correct reference links

2cc2f18

Signed-off-by: Zespre Chang <[email protected]>

starbops requested a review from w13915984028 December 17, 2025 03:49

starbops requested review from ihcsim and jillian-maroket December 17, 2025 03:49

brandboat reviewed Dec 17, 2025

View reviewed changes

docs/advanced/settings.md Outdated Show resolved Hide resolved

docs/advanced/settings.md Show resolved Hide resolved

brandboat reviewed Dec 17, 2025

View reviewed changes

docs/advanced/settings.md Outdated Show resolved Hide resolved

brandboat reviewed Dec 17, 2025

View reviewed changes

docs/advanced/settings.md Show resolved Hide resolved

starbops added 2 commits December 18, 2025 10:45

fix(upgrade): correct indentation for node upgrade options

e29fbda

Signed-off-by: Zespre Chang <[email protected]>

fix(upgrade): correct indentation for image preload options

2af29e8

Signed-off-by: Zespre Chang <[email protected]>

starbops requested a review from brandboat December 18, 2025 02:46

Merge branch 'main' into feat-8980

448270a

brandboat approved these changes Dec 18, 2025

View reviewed changes

jillian-maroket approved these changes Dec 18, 2025

View reviewed changes

starbops merged commit ba35e78 into harvester:main Dec 18, 2025
4 checks passed


		:::info

		Harvester applies the pause-relevant configuration during upgrade initialization. After that, any further changes to the fields under the `nodeUpgradeOption` option will not affect the current upgrade and can only take effect upon the next upgrade.


		- `nodeUpgradeOption`: Options for the node upgrading phase.

		In the node upgrade phase, Harvester upgrades each node's RKE2 and operating system autonomously. Harvester tries to migrate all virtual machines running on the to-be-upgraded node to other nodes. Harvester will shut down any virtual machine considered non-live-migratable to ensure the node upgrade goes smoothly. The `nodeUpgradeOption` provides customizability for how Harvester should behave upon node upgrades.

	- Pause for specific nodes: List the node names in the `pauseNodes` field. Nodes not included in the list are automatically upgraded.
	- Pause for specific nodes: When `mode` is set to `manual`, list the node names in the `pauseNodes` field. Nodes not included in the list are automatically upgraded. When `mode` is `auto`, the `pauseNodes` field is ignored.


		### Pausing Node Upgrades

		You can use the `nodeUpgradeOption` option in the [`upgrade-config`](advanced/settings.md#upgrade-config) setting to pause node upgrades.


		:::info important

		Upgrading of nodes listed in this field is _paused definitely_ until you take specific actions to [resume the process](upgrade/automatic.md#how-to-resume-a-node-to-continue-with-node-upgrade). Given that Harvester upgrades nodes sequentially, this implies that the entire upgrade progress is paused as well.

feat(upgrade): pausable node upgrade #924

feat(upgrade): pausable node upgrade #924

Uh oh!

Conversation

starbops commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem:

Solution:

Related Issue(s):

Test plan:

Additional documentation or context

Uh oh!

github-actions bot commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

w13915984028 left a comment

Choose a reason for hiding this comment

Uh oh!

w13915984028 Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

w13915984028 Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

starbops Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

ihcsim left a comment

Choose a reason for hiding this comment

Uh oh!

ihcsim Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

ihcsim Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

starbops Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

jillian-maroket left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brandboat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

starbops commented Nov 20, 2025 •

edited

Loading

github-actions bot commented Nov 20, 2025 •

edited

Loading