[BUG] [v1.1.3] Longhorn upgrade from v1.1.2 to v1.1.3-rc1 failed #3297
Description
Describe the bug
upgrading longhorn from v1.1.2 to v1.1.3-rc1 failed
To Reproduce
Steps to reproduce the behavior:
- install Longhorn v1.1.2
- upgrade to v1.1.3-rc1
- check longhorn-manager
Expected behavior
Longhorn upgrade should complete successfully.
Log or Support bundle
2021/11/22 19:20:22 proto: duplicate proto type registered: VersionResponse
time="2021-11-22T19:20:22Z" level=info msg="Start overwriting built-in settings with customized values"
W1122 19:20:22.179005 1 client_config.go:541] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
time="2021-11-22T19:20:22Z" level=info msg="cannot list the content of the src directory /var/lib/rancher/longhorn/engine-binaries for the copy, will do nothing: Failed to execute: nsenter [--mount=/host/proc/5384/ns/mnt --net=/host/proc/5384/ns/net bash -c ls /var/lib/rancher/longhorn/engine-binaries/*], output , stderr, ls: cannot access '/var/lib/rancher/longhorn/engine-binaries/*': No such file or directory\n, error exit status 2"
I1122 19:20:22.197560 1 leaderelection.go:241] attempting to acquire leader lease longhorn-system/longhorn-manager-upgrade-lock...
E1122 19:20:22.211098 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 1 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x181f780, 0x2a82e10)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
panic(0x181f780, 0x2a82e10)
/usr/local/go/src/runtime/panic.go:967 +0x166
k8s.io/client-go/tools/leaderelection/resourcelock.LeaseSpecToLeaderElectionRecord(...)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/client-go/tools/leaderelection/resourcelock/leaselock.go:108
k8s.io/client-go/tools/leaderelection/resourcelock.(*LeaseLock).Get(0xc000257a40, 0xc00004006a, 0x10, 0x43e09c)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/client-go/tools/leaderelection/resourcelock/leaselock.go:45 +0x18a
k8s.io/client-go/tools/leaderelection.(*LeaderElector).tryAcquireOrRenew(0xc00019d9e0, 0xe2)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:327 +0x13e
k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1()
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:243 +0x65
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00055e9a0)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00059e9a0, 0x77359400, 0x3ff3333333333333, 0x1, 0xc000190840)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire(0xc00019d9e0, 0x1d49ee0, 0xc000189ac0, 0x1d56700)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:242 +0x1ee
k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc00019d9e0, 0x1d49ee0, 0xc000189a40)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:201 +0x6b
k8s.io/client-go/tools/leaderelection.RunOrDie(0x1d49ee0, 0xc000189a40, 0x1d55aa0, 0xc000257a40, 0x4a817c800, 0x2540be400, 0x77359400, 0xc000189a80, 0xc00057d580, 0xc00057d5a0, ...)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:220 +0x96
github.com/longhorn/longhorn-manager/upgrade.upgrade(0xc00004006a, 0x10, 0xc00004002e, 0xf, 0xc0000ce800, 0xc0004051a0, 0xc0000fa000, 0x0, 0x0)
/go/src/github.com/longhorn/longhorn-manager/upgrade/upgrade.go:93 +0x350
github.com/longhorn/longhorn-manager/upgrade.Upgrade(0x0, 0x0, 0xc00004006a, 0x10, 0xc00006d9c0, 0xc0005c3e30)
/go/src/github.com/longhorn/longhorn-manager/upgrade/upgrade.go:70 +0x191
github.com/longhorn/longhorn-manager/app.startManager(0xc0000dda20, 0x1a5cd52, 0x0)
/go/src/github.com/longhorn/longhorn-manager/app/daemon.go:143 +0x4d0
github.com/longhorn/longhorn-manager/app.DaemonCmd.func1(0xc0000dda20)
/go/src/github.com/longhorn/longhorn-manager/app/daemon.go:73 +0x2f
github.com/urfave/cli.HandleAction(0x1778520, 0x1b22ad8, 0xc0000dda20, 0xc0000dda20, 0x0)
/go/src/github.com/longhorn/longhorn-manager/vendor/github.com/urfave/cli/app.go:525 +0x7c
github.com/urfave/cli.Command.Run(0x1a5dd85, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/longhorn/longhorn-manager/vendor/github.com/urfave/cli/command.go:174 +0x56e
github.com/urfave/cli.(*App).Run(0xc000252fc0, 0xc00018c000, 0xf, 0xf, 0x0, 0x0)
/go/src/github.com/longhorn/longhorn-manager/vendor/github.com/urfave/cli/app.go:276 +0x7ae
main.main()
/go/src/github.com/longhorn/longhorn-manager/main.go:67 +0x644
time="2021-11-22T19:20:22Z" level=info msg="Upgrade leader lost: meldafrawi-lh-01"
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x1523d9a]
goroutine 1 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
panic(0x181f780, 0x2a82e10)
/usr/local/go/src/runtime/panic.go:967 +0x166
k8s.io/client-go/tools/leaderelection/resourcelock.LeaseSpecToLeaderElectionRecord(...)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/client-go/tools/leaderelection/resourcelock/leaselock.go:108
k8s.io/client-go/tools/leaderelection/resourcelock.(*LeaseLock).Get(0xc000257a40, 0xc00004006a, 0x10, 0x43e09c)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/client-go/tools/leaderelection/resourcelock/leaselock.go:45 +0x18a
k8s.io/client-go/tools/leaderelection.(*LeaderElector).tryAcquireOrRenew(0xc00019d9e0, 0xe2)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:327 +0x13e
k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1()
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:243 +0x65
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00055e9a0)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00059e9a0, 0x77359400, 0x3ff3333333333333, 0x1, 0xc000190840)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire(0xc00019d9e0, 0x1d49ee0, 0xc000189ac0, 0x1d56700)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:242 +0x1ee
k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc00019d9e0, 0x1d49ee0, 0xc000189a40)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:201 +0x6b
k8s.io/client-go/tools/leaderelection.RunOrDie(0x1d49ee0, 0xc000189a40, 0x1d55aa0, 0xc000257a40, 0x4a817c800, 0x2540be400, 0x77359400, 0xc000189a80, 0xc00057d580, 0xc00057d5a0, ...)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:220 +0x96
github.com/longhorn/longhorn-manager/upgrade.upgrade(0xc00004006a, 0x10, 0xc00004002e, 0xf, 0xc0000ce800, 0xc0004051a0, 0xc0000fa000, 0x0, 0x0)
/go/src/github.com/longhorn/longhorn-manager/upgrade/upgrade.go:93 +0x350
github.com/longhorn/longhorn-manager/upgrade.Upgrade(0x0, 0x0, 0xc00004006a, 0x10, 0xc00006d9c0, 0xc0005c3e30)
/go/src/github.com/longhorn/longhorn-manager/upgrade/upgrade.go:70 +0x191
github.com/longhorn/longhorn-manager/app.startManager(0xc0000dda20, 0x1a5cd52, 0x0)
/go/src/github.com/longhorn/longhorn-manager/app/daemon.go:143 +0x4d0
github.com/longhorn/longhorn-manager/app.DaemonCmd.func1(0xc0000dda20)
/go/src/github.com/longhorn/longhorn-manager/app/daemon.go:73 +0x2f
github.com/urfave/cli.HandleAction(0x1778520, 0x1b22ad8, 0xc0000dda20, 0xc0000dda20, 0x0)
/go/src/github.com/longhorn/longhorn-manager/vendor/github.com/urfave/cli/app.go:525 +0x7c
github.com/urfave/cli.Command.Run(0x1a5dd85, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/longhorn/longhorn-manager/vendor/github.com/urfave/cli/command.go:174 +0x56e
github.com/urfave/cli.(*App).Run(0xc000252fc0, 0xc00018c000, 0xf, 0xf, 0x0, 0x0)
/go/src/github.com/longhorn/longhorn-manager/vendor/github.com/urfave/cli/app.go:276 +0x7ae
main.main()
/go/src/github.com/longhorn/longhorn-manager/main.go:67 +0x644
If applicable, add the Longhorn managers' log or support bundle when the issue happens.
You can generate a Support Bundle using the link at the footer of the Longhorn UI.
Environment
- Longhorn version: v1.1.3-rc1
- Installation method (e.g. Rancher Catalog App/Helm/Kubectl): kubectl
- Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: RKE
- Number of management node in the cluster: 1
- Number of worker node in the cluster: 3
- Node config
- OS type and version: Ubuntu
- CPU per node: 4
- Memory per node: 8 GB
- Disk type(e.g. SSD/NVMe): SSD
- Network bandwidth between the nodes:
- Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
- Number of Longhorn volumes in the cluster: 0
Additional context
Metadata
Metadata
Assignees
Labels
Type
Projects
Status