Describe the Bug

To Reproduce

On an unstable network try to rebuild a volume

Expected Behavior

It should be impossible for replica rebuild progress to go over 100%

Support Bundle for Troubleshooting

I will try to reproduce again and provide a bundle. For now I assume the one from here could be used: #9626 (comment)

Environment

• Longhorn version: 1.11.1
• Impacted volume (PV):
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl): helm
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: Bare metal k8s
  • Number of control plane nodes in the cluster: 1
  • Number of worker nodes in the cluster: 2
• Node config
  • OS type and version:
  • Kernel version:
  • CPU per node:
  • Memory per node:
  • Disk type (e.g. SSD/NVMe/HDD):
  • Network bandwidth between the nodes (Gbps):
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): KVM
• Number of Longhorn volumes in the cluster:

Additional context

I assume I'm seeing this because my nodes use Nebula VPN to communicate, which seems to be causing the network to be inconsistent.

Workaround and Mitigation

You can use disaster recovery backups to sync the volume to other nodes, then delete the faulty volume and replace it with the disaster recovery volumes.