handle nil pointer dereference and added a test #14145
Conversation
|
Hi @ayushpatil2122. Thanks for your PR. PRs from untrusted users cannot be marked as trusted with I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
pkg/virt-handler/options_test.go
Outdated
| It("should panic when NUMA cells are nil", func() { | ||
| caps := &libvirtxml.Caps{Host: libvirtxml.CapsHost{NUMA: &libvirtxml.CapsHostNUMATopology{}}} | ||
| caps.Host.NUMA.Cells = nil | ||
|
|
||
| Expect(func() { | ||
| capabilitiesToTopology(caps) | ||
| }).To(Panic()) | ||
| }) | ||
|
|
||
| It("should panic when NUMA cell memory is nil", func() { | ||
| caps := &libvirtxml.Caps{Host: libvirtxml.CapsHost{NUMA: &libvirtxml.CapsHostNUMATopology{}}} | ||
| caps.Host.NUMA.Cells = &libvirtxml.CapsHostNUMACells{ | ||
| Cells: []libvirtxml.CapsHostNUMACell{ | ||
| { | ||
| Memory: nil, // This should cause a panic | ||
| PageInfo: []libvirtxml.CapsHostNUMAPageInfo{ | ||
| {Unit: memoryUnit, Size: 4, Count: 4064224}, | ||
| }, | ||
| Distances: &libvirtxml.CapsHostNUMADistances{Siblings: []libvirtxml.CapsHostNUMASibling{{ID: 0, Value: 10}}}, | ||
| CPUS: &libvirtxml.CapsHostNUMACPUs{CPUs: []libvirtxml.CapsHostNUMACPU{ | ||
| {ID: 0, Siblings: "0,4"}, | ||
| }}, | ||
| }, | ||
| }, | ||
| } | ||
|
|
||
| Expect(func() { | ||
| capabilitiesToTopology(caps) | ||
| }).To(Panic()) | ||
| }) | ||
|
|
||
| It("should panic when NUMA cell CPUs are nil", func() { | ||
| caps := &libvirtxml.Caps{Host: libvirtxml.CapsHost{NUMA: &libvirtxml.CapsHostNUMATopology{}}} | ||
| caps.Host.NUMA.Cells = &libvirtxml.CapsHostNUMACells{ | ||
| Cells: []libvirtxml.CapsHostNUMACell{ | ||
| { | ||
| Memory: &libvirtxml.CapsHostNUMAMemory{Unit: memoryUnit, Size: 16256896}, | ||
| PageInfo: []libvirtxml.CapsHostNUMAPageInfo{ | ||
| {Unit: memoryUnit, Size: 4, Count: 4064224}, | ||
| }, | ||
| Distances: &libvirtxml.CapsHostNUMADistances{Siblings: []libvirtxml.CapsHostNUMASibling{{ID: 0, Value: 10}}}, | ||
| CPUS: nil, // This should cause a panic | ||
| }, | ||
| }, | ||
| } | ||
|
|
||
| Expect(func() { | ||
| capabilitiesToTopology(caps) | ||
| }).To(Panic()) | ||
| }) | ||
|
|
||
| It("should panic when NUMA cell distances are nil", func() { | ||
| caps := &libvirtxml.Caps{Host: libvirtxml.CapsHost{NUMA: &libvirtxml.CapsHostNUMATopology{}}} | ||
| caps.Host.NUMA.Cells = &libvirtxml.CapsHostNUMACells{ | ||
| Cells: []libvirtxml.CapsHostNUMACell{ | ||
| { | ||
| Memory: &libvirtxml.CapsHostNUMAMemory{Unit: memoryUnit, Size: 16256896}, | ||
| PageInfo: []libvirtxml.CapsHostNUMAPageInfo{ | ||
| {Unit: memoryUnit, Size: 4, Count: 4064224}, | ||
| }, | ||
| Distances: nil, // This should cause a panic | ||
| CPUS: &libvirtxml.CapsHostNUMACPUs{CPUs: []libvirtxml.CapsHostNUMACPU{ | ||
| {ID: 0, Siblings: "0,4"}, | ||
| }}, | ||
| }, | ||
| }, | ||
| } | ||
|
|
||
| Expect(func() { | ||
| capabilitiesToTopology(caps) | ||
| }).To(Panic()) | ||
| }) | ||
|
|
There was a problem hiding this comment.
You can merge this into a DescribeTable and pass the various struct as arguments
There was a problem hiding this comment.
As @alicefr suggested, this can be:
DescribeTable("should not panic", func(cells *libvirtxml.CapsHostNUMACells) {
caps := &libvirtxml.Caps{Host: libvirtxml.CapsHost{NUMA: &libvirtxml.CapsHostNUMATopology{}}}
caps.Host.NUMA.Cells = cells
Expect(func() {
capabilitiesToTopology(caps)
}).ToNot(Panic())
},
Entry("when NUMA cells are nil", nil),
Entry("when NUMA cell memory is nil", &libvirtxml.CapsHostNUMACells{
Cells: []libvirtxml.CapsHostNUMACell{
{
Memory: nil, // This should cause a panic
PageInfo: []libvirtxml.CapsHostNUMAPageInfo{
{Unit: memoryUnit, Size: 4, Count: 4064224},
},
Distances: &libvirtxml.CapsHostNUMADistances{Siblings: []libvirtxml.CapsHostNUMASibling{{ID: 0, Value: 10}}},
CPUS: &libvirtxml.CapsHostNUMACPUs{CPUs: []libvirtxml.CapsHostNUMACPU{
{ID: 0, Siblings: "0,4"},
}},
},
}}),
Entry("when NUMA cell CPUs are nil", &libvirtxml.CapsHostNUMACells{
Cells: []libvirtxml.CapsHostNUMACell{
{
Memory: &libvirtxml.CapsHostNUMAMemory{Unit: memoryUnit, Size: 16256896},
PageInfo: []libvirtxml.CapsHostNUMAPageInfo{
{Unit: memoryUnit, Size: 4, Count: 4064224},
},
Distances: &libvirtxml.CapsHostNUMADistances{Siblings: []libvirtxml.CapsHostNUMASibling{{ID: 0, Value: 10}}},
CPUS: nil, // This should cause a panic
},
},
}),
Entry("panic when NUMA cell distances are nil", &libvirtxml.CapsHostNUMACells{
Cells: []libvirtxml.CapsHostNUMACell{
{
Memory: &libvirtxml.CapsHostNUMAMemory{Unit: memoryUnit, Size: 16256896},
PageInfo: []libvirtxml.CapsHostNUMAPageInfo{
{Unit: memoryUnit, Size: 4, Count: 4064224},
},
Distances: nil, // This should cause a panic
CPUS: &libvirtxml.CapsHostNUMACPUs{CPUs: []libvirtxml.CapsHostNUMACPU{
{ID: 0, Siblings: "0,4"},
}},
},
},
}),
)NOTE
We want this NOT to panic, the assertion and test description were wrong
|
@ayushpatil2122 I will repeat here my previous comments, can you please verify that this actually fixes the original issue #14064 . Otherwise, we don't actually know if this fix the issue |
|
Also as @alicefr mentioned were you able to check if it actually occurs by testing it out in minikube? |
|
Yes, please approve this. The cluster I'm working on is facing the same issues, and hopefully this will fix it 🤞 |
|
If the segfault is due to configuration options (AFAIR) it should be possible to create a reproducer with the test framework. You could do something like:
This is hitting a lot of users so please do let us know if you don't have the time to work on it. |
I written a unit test as well to reproduce this issue |
|
Any chance this will get backported into the 1.5 release once merged? I can't test 1.5 locally without a full server environment. Edit: I cloned down this branch and tested it with my K3D cluster and it solves my issue. I haven't tested with Minkube, but I had the same stack trace so I think this PR looks good |
|
@victortoso are there plans for this to get merged? I have 200/420 nodes breaking with the same errors. Can you push this through, please 🙏? |
pkg/virt-handler/options.go
Outdated
| func capabilitiesToTopology(capabilities *libvirtxml.Caps) *cmdv1.Topology { | ||
| topology := &cmdv1.Topology{} | ||
| if capabilities == nil { | ||
| if capabilities == nil || capabilities.Host.NUMA == nil { |
There was a problem hiding this comment.
My feeling about this patch is that it does not solve the problem. It just adds guards to avoid a segfault.
It wasn't nil before, why it is now? When did this happen?
The guards aren't wrong and the patch might as well avoid a segfault but I would love to know why it became a problem now.
There was a problem hiding this comment.
This commit looks like the culprit.
5cb6bd3.
Initial thoughts are there is a difference in how the XML is being unmarshaled with the switch to libvirt. I will try cherry picking this commit to 1.3 since the issue does not appear on that branch and let you know if the error reproduces itself. I will update this post as I learn more.
|
Alright, my previous thought about this commit being the issue seems to be correct. @victortoso I compared the In the commit above, options.go now uses libvirts XML model as opposed to the one located here https://github.com/kubevirt/kubevirt/blob/v1.3.0/pkg/virt-handler/node-labeller/api/capabilities.go. You can see the distances struct is an slice of siblings but not a pointer. In libvirts model it is a pointer to a structure, not an slice. See https://pkg.go.dev/github.com/libvirt/libvirt-go-xml#CapsHostNUMACell. In the XML that I provided above there is no Since libvirt uses a pointer instead of an slice, the structure will be nil when the distances entry does not exist during unmarshaling. Previously, distances were an slice, so when unmarshaled, it would be initialized as empty, preventing a segmentation fault during the loop. Some of the nil checks in this PR may be redundant as the only line I had to touch was options.go:98 in order to fix the segfault. However, I think adding the extra nil checks is a good idea since the pointers could potentially be nil. As a sanity check to make sure this commit is the problem I also cherry picked it and applied it to Kubevirt 1.3, a version before this change was introduced. I was able to reproduce the same stack trace. stacktrace (Kubevirt 1.3 with commit 5cb6bd3): |
Many thanks for the deep analysis. |
fossedihelm
left a comment
There was a problem hiding this comment.
Thank you!
Please see my comments below.
Do not forget to run make && make generate before pushing changes.
pkg/virt-handler/options_test.go
Outdated
| It("should panic when NUMA cells are nil", func() { | ||
| caps := &libvirtxml.Caps{Host: libvirtxml.CapsHost{NUMA: &libvirtxml.CapsHostNUMATopology{}}} | ||
| caps.Host.NUMA.Cells = nil | ||
|
|
||
| Expect(func() { | ||
| capabilitiesToTopology(caps) | ||
| }).To(Panic()) | ||
| }) | ||
|
|
||
| It("should panic when NUMA cell memory is nil", func() { | ||
| caps := &libvirtxml.Caps{Host: libvirtxml.CapsHost{NUMA: &libvirtxml.CapsHostNUMATopology{}}} | ||
| caps.Host.NUMA.Cells = &libvirtxml.CapsHostNUMACells{ | ||
| Cells: []libvirtxml.CapsHostNUMACell{ | ||
| { | ||
| Memory: nil, // This should cause a panic | ||
| PageInfo: []libvirtxml.CapsHostNUMAPageInfo{ | ||
| {Unit: memoryUnit, Size: 4, Count: 4064224}, | ||
| }, | ||
| Distances: &libvirtxml.CapsHostNUMADistances{Siblings: []libvirtxml.CapsHostNUMASibling{{ID: 0, Value: 10}}}, | ||
| CPUS: &libvirtxml.CapsHostNUMACPUs{CPUs: []libvirtxml.CapsHostNUMACPU{ | ||
| {ID: 0, Siblings: "0,4"}, | ||
| }}, | ||
| }, | ||
| }, | ||
| } | ||
|
|
||
| Expect(func() { | ||
| capabilitiesToTopology(caps) | ||
| }).To(Panic()) | ||
| }) | ||
|
|
||
| It("should panic when NUMA cell CPUs are nil", func() { | ||
| caps := &libvirtxml.Caps{Host: libvirtxml.CapsHost{NUMA: &libvirtxml.CapsHostNUMATopology{}}} | ||
| caps.Host.NUMA.Cells = &libvirtxml.CapsHostNUMACells{ | ||
| Cells: []libvirtxml.CapsHostNUMACell{ | ||
| { | ||
| Memory: &libvirtxml.CapsHostNUMAMemory{Unit: memoryUnit, Size: 16256896}, | ||
| PageInfo: []libvirtxml.CapsHostNUMAPageInfo{ | ||
| {Unit: memoryUnit, Size: 4, Count: 4064224}, | ||
| }, | ||
| Distances: &libvirtxml.CapsHostNUMADistances{Siblings: []libvirtxml.CapsHostNUMASibling{{ID: 0, Value: 10}}}, | ||
| CPUS: nil, // This should cause a panic | ||
| }, | ||
| }, | ||
| } | ||
|
|
||
| Expect(func() { | ||
| capabilitiesToTopology(caps) | ||
| }).To(Panic()) | ||
| }) | ||
|
|
||
| It("should panic when NUMA cell distances are nil", func() { | ||
| caps := &libvirtxml.Caps{Host: libvirtxml.CapsHost{NUMA: &libvirtxml.CapsHostNUMATopology{}}} | ||
| caps.Host.NUMA.Cells = &libvirtxml.CapsHostNUMACells{ | ||
| Cells: []libvirtxml.CapsHostNUMACell{ | ||
| { | ||
| Memory: &libvirtxml.CapsHostNUMAMemory{Unit: memoryUnit, Size: 16256896}, | ||
| PageInfo: []libvirtxml.CapsHostNUMAPageInfo{ | ||
| {Unit: memoryUnit, Size: 4, Count: 4064224}, | ||
| }, | ||
| Distances: nil, // This should cause a panic | ||
| CPUS: &libvirtxml.CapsHostNUMACPUs{CPUs: []libvirtxml.CapsHostNUMACPU{ | ||
| {ID: 0, Siblings: "0,4"}, | ||
| }}, | ||
| }, | ||
| }, | ||
| } | ||
|
|
||
| Expect(func() { | ||
| capabilitiesToTopology(caps) | ||
| }).To(Panic()) | ||
| }) | ||
|
|
There was a problem hiding this comment.
As @alicefr suggested, this can be:
DescribeTable("should not panic", func(cells *libvirtxml.CapsHostNUMACells) {
caps := &libvirtxml.Caps{Host: libvirtxml.CapsHost{NUMA: &libvirtxml.CapsHostNUMATopology{}}}
caps.Host.NUMA.Cells = cells
Expect(func() {
capabilitiesToTopology(caps)
}).ToNot(Panic())
},
Entry("when NUMA cells are nil", nil),
Entry("when NUMA cell memory is nil", &libvirtxml.CapsHostNUMACells{
Cells: []libvirtxml.CapsHostNUMACell{
{
Memory: nil, // This should cause a panic
PageInfo: []libvirtxml.CapsHostNUMAPageInfo{
{Unit: memoryUnit, Size: 4, Count: 4064224},
},
Distances: &libvirtxml.CapsHostNUMADistances{Siblings: []libvirtxml.CapsHostNUMASibling{{ID: 0, Value: 10}}},
CPUS: &libvirtxml.CapsHostNUMACPUs{CPUs: []libvirtxml.CapsHostNUMACPU{
{ID: 0, Siblings: "0,4"},
}},
},
}}),
Entry("when NUMA cell CPUs are nil", &libvirtxml.CapsHostNUMACells{
Cells: []libvirtxml.CapsHostNUMACell{
{
Memory: &libvirtxml.CapsHostNUMAMemory{Unit: memoryUnit, Size: 16256896},
PageInfo: []libvirtxml.CapsHostNUMAPageInfo{
{Unit: memoryUnit, Size: 4, Count: 4064224},
},
Distances: &libvirtxml.CapsHostNUMADistances{Siblings: []libvirtxml.CapsHostNUMASibling{{ID: 0, Value: 10}}},
CPUS: nil, // This should cause a panic
},
},
}),
Entry("panic when NUMA cell distances are nil", &libvirtxml.CapsHostNUMACells{
Cells: []libvirtxml.CapsHostNUMACell{
{
Memory: &libvirtxml.CapsHostNUMAMemory{Unit: memoryUnit, Size: 16256896},
PageInfo: []libvirtxml.CapsHostNUMAPageInfo{
{Unit: memoryUnit, Size: 4, Count: 4064224},
},
Distances: nil, // This should cause a panic
CPUS: &libvirtxml.CapsHostNUMACPUs{CPUs: []libvirtxml.CapsHostNUMACPU{
{ID: 0, Siblings: "0,4"},
}},
},
},
}),
)NOTE
We want this NOT to panic, the assertion and test description were wrong
pkg/virt-handler/options.go
Outdated
| if cell.PageInfo != nil { | ||
| for _, page := range cell.PageInfo { | ||
| c.Pages = append(c.Pages, pageToPage(page)) | ||
| } |
There was a problem hiding this comment.
iterate over nil slice is a safe non-op. Please drop this
pkg/virt-handler/options.go
Outdated
|
|
||
| for _, distance := range cell.Distances.Siblings { | ||
| c.Distances = append(c.Distances, distanceToDistance(distance)) | ||
| if cell.Distances != nil && cell.Distances.Siblings != nil { |
There was a problem hiding this comment.
iterate over nil slice is a safe non-op:
| if cell.Distances != nil && cell.Distances.Siblings != nil { | |
| if cell.Distances != nil { |
pkg/virt-handler/options.go
Outdated
|
|
||
| for _, cpu := range cell.CPUS.CPUs { | ||
| c.Cpus = append(c.Cpus, cpuToCPU(cpu)) | ||
| if cell.CPUS != nil && cell.CPUS.CPUs != nil { |
There was a problem hiding this comment.
iterate over nil slice is a safe non-op:
| if cell.CPUS != nil && cell.CPUS.CPUs != nil { | |
| if cell.CPUS != nil { |
pkg/virt-handler/options.go
Outdated
| func capabilitiesToTopology(capabilities *libvirtxml.Caps) *cmdv1.Topology { | ||
| topology := &cmdv1.Topology{} | ||
| if capabilities == nil { | ||
| if capabilities == nil || capabilities.Host.NUMA == nil { |
There was a problem hiding this comment.
Please use early exit:
| if capabilities == nil || capabilities.Host.NUMA == nil { | |
| if capabilities == nil || capabilities.Host.NUMA == nil || capabilities.Host.NUMA.Cells == nil |
and drop the changes at line 78
|
@ayushpatil2122 Please squash the two commits into one. |
6d55d51 to
bb672e0
Compare
victortoso
left a comment
There was a problem hiding this comment.
/test pull-kubevirt-build
/test pull-kubevirt-code-lint
/test pull-kubevirt-unit-test
pkg/virt-handler/options.go
Outdated
| for _, page := range cell.PageInfo { | ||
| c.Pages = append(c.Pages, pageToPage(page)) | ||
| c.Pages = append(c.Pages, pageToPage(page)) |
Let me know if you hit any issues. |
30baa4d to
e1a5ec1
Compare
|
You are almost there. Let's try to get it done in the next iteration so we can move on to merge and backport this. I've fixed the build and pushed to my review-14145 branch. You need to squash this commit to your PR. Please do @ mention me so I can take a look as soon as you get this done. Thanks again for your work. |
Signed-off-by: ayushpatil2122 <[email protected]> Signed-off-by: ayushpatil2122 <[email protected]> Signed-off-by: ayushpatil2122 <[email protected]> Signed-off-by: ayushpatil2122 <[email protected]> Signed-off-by: ayushpatil2122 <[email protected]>
3aec2a4 to
078f7df
Compare
|
@victortoso done thanks to you |
|
/test pull-kubevirt-build |
|
/lgtm Thanks again! |
|
Required labels detected, running phase 2 presubmits: |
|
/retest-required |
|
I missed the fact that we should have added the history behind this change into the commit log. My bad. The commit that introduced this issue 5cb6bd3 and is present in both v1.4.0 and v1.5.0 so we should backport it. I'm not sure I can do it, but let's try. /cherry-pick release-1.5 |
|
@victortoso: new pull request created: #14402 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@victortoso: new pull request created: #14403 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@victortoso and @ayushpatil2122 thank you very much for pushing through! Without trying to put more pressure... any idea when this gets released? 😇 |
Not sure but soon |
@max06 this PR is target for kubevirt 1.6 release, several weeks away. |
|
@victortoso good morning and thanks for the fast reply! I tried setting up a fresh installation with v1.5.0 and ran into this exact issue. To me it looks like it got backported, but not released yet in a v1.5.1 for example. |
|
@max06 Exactly! We are discussing it. Will let you know! |
|
The plan is to create a z-release during this week. |
|
Thank you very much! 🙇🏼♂️ |
Any updates? |
|
Apologies, it was re-scheduled for Monday, 05 |
What this PR does
Before this PR:
virt-handler became CrashLoopBackOff due to invalid memory address or nil pointer dereference
After this PR:
handled the nil pointer dereference
Fixes #14064
Why we need it and why it was done in this way
The following tradeoffs were made:
The following alternatives were considered:
Links to places where the discussion took place:
Special notes for your reviewer
Checklist
This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.
Release note