Conversation
Rather than each device implementation holding a reference to the guest memory move this to VhostUserCommon. This refactoring simplifies the function signatures but also allows for methods that act on VhostUserCommon that don't have the memory available to them. Signed-off-by: Rob Bradford <[email protected]>
Query and save the dirty log before shutting down the vhost-user device. This allows any final dirty memory ranges to be recorded before it becomes impossible to do that as the vhost-user handle has been closed. This is required to ensure that all memory writes have been correctly recorded that may be triggered by inflight I/O drains from vhost-user device state capture. One small implementation wrinkle: with local migrations there is no dirty logging (since we just pass the memory FD over the socket) so calling dirty_log() would generate an error. As there is no clean way to query if dirty logging has been started add a boolean to track if its active. Signed-off-by: Rob Bradford <[email protected]>
During migration send one final set of changed memory after capturing the snapshot/state. This captures any memory changed as a side effect of capturing that state. In particular with vhost-user capturing the device state can lead to inflight requests being drained/flushed which could change memory. As this is related to the snapshot account for this memory transfer in the snapshot metrics. No equivalent change is needed for snapshot as the memory is written after the state is snapshotted. Signed-off-by: Rob Bradford <[email protected]>
ba13875 to
1864ca7
Compare
|
This took me a bit to understand, but TLDR, LGTM. The issue is that pause is a pure pause(vring only) and doesn't stop the device(I/O continues). So the actual synchronous work stoppage ended up as part of snapshot. I agree having these be asymmetrical smells. I feel like we're missing a state. pause is insufficient for snapshot, so currently there is a vhost-user stop tacked on to snapshot and pre-pause for block. What if we added an explicit step to flush/quiesce I/O pre-snapshot? |
| if self.migration_started { | ||
| // Local migration does not enable dirty logging. | ||
| if self.dirty_logging { | ||
| self.saved_dirty_log = Some(self.dirty_log()?); |
There was a problem hiding this comment.
That's a clever trick given we don't have access to the vhost-user device after it's been shutdown.
One other approach I can think of would be to update the shutdown() method to return Result<Option<MemoryRangeTable>>, but maybe that's not ideal as it would tie shutdown to migration semantics.
There was a problem hiding this comment.
Indeed - I thought this elegant but not too "clever" (in terms of being hard to understand).
|
|
||
| // One final memory iteration to handle side effects from snapshot. | ||
| if !send_data_migration.local { | ||
| let memory_ranges = vm.dirty_log()?; |
There was a problem hiding this comment.
I suppose the amount of memory here will always be fairly negligible?
7042922
When taking the snapshot of a vhost-user device this can cause the backend to
force a drain of inflight I/O which could result in guest memory changes. Carry
out one more memory transfer after capturing the state but before sending it
since the protocol does not tolerate a memory update after the state has been
received.