block: qcow: Implement QCOW2 v3 corrupt bit support - #7639
Conversation
phip1611
left a comment
There was a problem hiding this comment.
Great contribution! Left some remarks
| .unwrap(); | ||
| let features = disk_file.read_u64::<BigEndian>().unwrap(); | ||
| assert_ne!( | ||
| features & IncompatFeatures::CORRUPT.bits(), |
There was a problem hiding this comment.
nit: assert_eq!(IncompatFeatures::from_bits_retain(features), IncompatFeatures::CORRUPT)
There was a problem hiding this comment.
Yeah, fixed this with assert!(IncompatFeatures::from_bits_retain(features).contains(IncompatFeatures::CORRUPT)), as some other bits might be set, too. Thanks
| @@ -1562,8 +1562,12 @@ impl QcowFile { | |||
| let mut decompressed_cluster = vec![0; cluster_size]; | |||
| let decompressed_size = decoder | |||
| .decode(&compressed_cluster, &mut decompressed_cluster) | |||
| .map_err(|_| std::io::Error::from_raw_os_error(EIO))?; | |||
| .map_err(|_| { | |||
There was a problem hiding this comment.
not about this line but the commit message
Note: Marking decompression failures as corrupt is more conservative
than QEMU, which returns EIO without setting the corrupt bit. This is
debatable since corrupted compressed data doesn't necessarily indicate
metadata corruption, but it provides a stronger safety guarantee by
preventing further writes to potentially damaged images.
I think this is good, sounds reasonable!
There was a problem hiding this comment.
Gotcha, thanks for confirming!
| .map_err(|_| std::io::Error::from_raw_os_error(EIO))?; | ||
| .map_err(|_| { | ||
| let _ = self.header.set_corrupt_bit(self.raw_file.file_mut()); | ||
| std::io::Error::from_raw_os_error(EIO) |
There was a problem hiding this comment.
nit: io::Error::from_raw_os_error(EIO) and use std::io in the top
There was a problem hiding this comment.
Touched this one. Looks like it already imports std::io::{self, ... at the top. To note is - this line is just moved around and I think there are numerous other occurrences of the same across the file. We can sure plan for a little style cleanups. The occurrences that have been introduced or touched in this PR should be fixed, though. thanks
| @@ -2000,6 +2005,7 @@ impl QcowFile { | |||
| return Err(e); | |||
| } | |||
| Err(refcount::Error::InvalidIndex) => { | |||
| let _ = self.header.set_corrupt_bit(self.raw_file.file_mut()); | |||
There was a problem hiding this comment.
I'm pretty sure let _ = is not necessary for Result<()>- clippy should not complain
There was a problem hiding this comment.
Changed to .ok(), maybe slightly more idiomatic. Please let me know, if there's a better option.
Just removing let _ = doesn't compile - Result has #[must_use], so clippy won't pass.
The result is ignored here because returning the actual error (EIO/EINVAL) to the caller is more important than whether the corrupt bit was successfully written. The corrupt bit is a best effort safeguard, but it shouldn't mask or interfere with the primary error.
Thanks
There was a problem hiding this comment.
Ahh yes, of course, missed that! then I guess .unwrap() or .expect() is the way to go or just ? to propagte the error
There was a problem hiding this comment.
The corrupt bit is best effort. If it can't be written (disk full, hardware error, readonly storage, etc.), the actual corruption error (EIO) still has to be returned to the caller. Using ? would mask the corruption error with a potentially unrelated I/O error, and .unwrap() / .expect() would panic which seems too aggressive for a safeguard mechanism.
Thus, in this case I'd rather agree with QEMU - it handles it the same way by ignoring the return value when setting the corrupt bit. The initial let _ = intention, where .ok() is semantically equivalent.
Please see here the excerpts:
if (fatal) {
qcow2_mark_corrupt(bs); // Return value ignored!
bs->drv = NULL; /* make BDS unusable */
}And qcow2_mark_corrupt returns int:
int qcow2_mark_corrupt(BlockDriverState *bs)
{
BDRVQcow2State *s = bs->opaque;
s->incompatible_features |= QCOW2_INCOMPAT_CORRUPT;
return qcow2_update_header(bs); // Can fail!
}Maybe there's a better mechanism in Rust to handle such a situation?
Thanks
There was a problem hiding this comment.
Maybe this?
if let Err(e) = self.header.set_corrupt_bit(&mut file) {
debug!("Failed to persist corrupt bit: {e}");
}Otherwise there wouldn't seem to be a "magic" Rust mechanism that would be fundamentally better.
Thanks
| @@ -2030,6 +2061,10 @@ impl QcowFile { | |||
| Err(refcount::Error::RefcountOverflow { .. }) => { | |||
| return Err(std::io::Error::from_raw_os_error(EINVAL)); | |||
| } | |||
| Err(refcount::Error::RefblockUnaligned(_)) => { | |||
| let _ = self.header.set_corrupt_bit(self.raw_file.file_mut()); | |||
| return Err(std::io::Error::from_raw_os_error(EIO)); | |||
There was a problem hiding this comment.
same here, please just io::Error::from_raw_os_error(EIO) + use std::io
| let cluster_size = self.raw_file.cluster_size(); | ||
| if l2_addr_disk & (cluster_size - 1) != 0 { | ||
| let _ = self.header.set_corrupt_bit(self.raw_file.file_mut()); | ||
| return Err(std::io::Error::from_raw_os_error(EIO)); |
There was a problem hiding this comment.
same here, please just io::Error::from_raw_os_error(EIO) + use std::io
| fn set_corrupt_flag(path: &std::path::Path, corrupt: bool) -> std::io::Result<()> { | ||
| let mut file = OpenOptions::new().read(true).write(true).open(path)?; | ||
|
|
||
| file.seek(SeekFrom::Start(72))?; |
There was a problem hiding this comment.
the magic 72 could be replaced by const MAGIC: usize = 72;. Please use a descriptive name
| Ok(Some(info)) | ||
| } | ||
|
|
||
| fn check_dirty_flag(path: &std::path::Path) -> Result<Option<bool>, String> { |
There was a problem hiding this comment.
I'd prefer to have simply &Path here. Further, the most idiomatic thing in Rust for code consuming paths is to use path: impl AsRef<Path>. Example in the standard library .
I'm however also fine with &Path, but please no fully qualified paths. Dependencies of a module should be visible in the top from the use block
There was a problem hiding this comment.
Using &Path here.
I have to mention, at the time I started coding Rust having rustfmt seemed amazing to just have one style that fits them all and focus on the actual coding. But seems there are still things that rustfmt doesn't cover. Maybe there are other tools like astyle for C++ and alike, that could solve this even more effectively :)
Thanks
| .spawn() | ||
| .unwrap(); | ||
|
|
||
| std::thread::sleep(std::time::Duration::from_secs(5)); |
There was a problem hiding this comment.
please just sleep() and Duration with corresponding use statements in the top
if the other integration tests also also do it like this - fine with me, no need to clean this up right now.
There was a problem hiding this comment.
thread::sleep(Duration .... seems the shortest I see right now. In general, yep, seems quite some coding styles in here. Thanks
dcf7254 to
c8c59cf
Compare
| if is_writable { | ||
| return Err(Error::CorruptImage); | ||
| } | ||
| log::warn!("QCOW2 image is marked corrupt, opening read-only"); |
There was a problem hiding this comment.
For consistency please import warn and use warn!() here
| if decompressed_size as u64 != self.raw_file.cluster_size() { | ||
| self.header.set_corrupt_bit(self.raw_file.file_mut()).ok(); |
There was a problem hiding this comment.
Can we use self.header.set_corrupt_bit(self.raw_file.file_mut())?;
There was a problem hiding this comment.
Hi @russell-islam, thanks for the review!
This is intentional - please see the discussion above with @phip1611 about this: #7639 (comment)
In short - the corrupt bit is best effort. Using ? would mask the actual corruption error (EIO) with a potentially unrelated I/O error if writing the corrupt bit fails. QEMU handles it the same way by ignoring the return value.
I suggested an alternative with logging:
if let Err(e) = self.header.set_corrupt_bit(&mut file) {
debug!("Failed to persist corrupt bit: {e}");
}If there's a better approach, please let me know.
Thanks!
There was a problem hiding this comment.
I think a warn! would be better here. I like this approach, sounds sensible!
There was a problem hiding this comment.
Thanks! Went with this approach and added set_corrupt_bit_best_effort helper method throwing a warn!. This way we're arguably handling it better than QEMU. Platform ops get visibility into these edge cases through logs and telemetry, while still preserving the original error for the caller.
| let start = l2_entry_std_cluster_addr(l2_entry) + self.raw_file.cluster_offset(address); | ||
| let cluster_addr = l2_entry_std_cluster_addr(l2_entry); | ||
| if cluster_addr & (self.raw_file.cluster_size() - 1) != 0 { | ||
| self.header.set_corrupt_bit(self.raw_file.file_mut()).ok(); |
There was a problem hiding this comment.
Same comment as above.
| .map_err(|e| std::io::Error::other(Error::GettingRefcount(e)))?; | ||
| .map_err(|e| { | ||
| if matches!(e, refcount::Error::RefblockUnaligned(_)) { | ||
| self.header.set_corrupt_bit(self.raw_file.file_mut()).ok(); |
| l2_entry_std_cluster_addr(l2_entry) | ||
| let cluster_addr = l2_entry_std_cluster_addr(l2_entry); | ||
| if cluster_addr & (self.raw_file.cluster_size() - 1) != 0 { | ||
| self.header.set_corrupt_bit(self.raw_file.file_mut()).ok(); |
| @@ -1717,6 +1765,10 @@ impl QcowFile { | |||
| fn get_new_cluster(&mut self, initial_data: Option<Vec<u8>>) -> std::io::Result<u64> { | |||
| // First use a pre allocated cluster if one is available. | |||
| if let Some(free_cluster) = self.avail_clusters.pop() { | |||
| if free_cluster == 0 { | |||
| self.header.set_corrupt_bit(self.raw_file.file_mut()).ok(); | |||
| @@ -1727,6 +1779,10 @@ impl QcowFile { | |||
|
|
|||
| let max_valid_cluster_offset = self.refcounts.max_valid_cluster_offset(); | |||
| if let Some(new_cluster) = self.raw_file.add_cluster_end(max_valid_cluster_offset)? { | |||
| if new_cluster == 0 { | |||
| self.header.set_corrupt_bit(self.raw_file.file_mut()).ok(); | |||
| @@ -1837,6 +1893,9 @@ impl QcowFile { | |||
| .refcounts | |||
| .get_cluster_refcount(&mut self.raw_file, cluster_addr) | |||
| .map_err(|e| { | |||
| if matches!(e, refcount::Error::RefblockUnaligned(_)) { | |||
| self.header.set_corrupt_bit(self.raw_file.file_mut()).ok(); | |||
| @@ -20,6 +20,9 @@ pub enum Error { | |||
| /// `InvalidIndex` - Address requested isn't within the range of the disk. | |||
| #[error("Address requested is not within the range of the disk")] | |||
| InvalidIndex, | |||
| /// `RefblockUnaligned` - Refcount block offset is not cluster aligned. | |||
| #[error("Refcount block offset {0:#x} is not cluster aligned")] | |||
| RefblockUnaligned(u64), | |||
There was a problem hiding this comment.
RefBlockUnAllocated?
There was a problem hiding this comment.
The error refers to an alignment issue, not allocation. The refcount block offset is not cluster-aligned as offset & (cluster_size - 1) != 0.
RefBlockUnAllocated would imply the block doesn't exist, which is a different condition.
Thanks
There was a problem hiding this comment.
More detail on this - the L1 table, L2 tables, and refcount table all contain offsets that must be cluster aligned per the QCOW2 spec. When looking up a guest cluster address, the L1 entry points to an L2 table, and the L2 entry points to the data cluster. Similarly, when tracking cluster usage, the refcount table points to refcount blocks. All of these offsets must satisfy offset & (cluster_size - 1) == 0.
These are new validations added in this PR. If any of these offsets fail the alignment check, the metadata is corrupt, indicating either disk corruption, a buggy QCOW2 implementation, et cetera. This is what RefblockUnaligned and the similar L2 alignment checks detect.
An unallocated entry would be zero, which is a normal condition handled separately and doesn't indicate corruption.
Thanks
2ca15ed to
af8c074
Compare
phip1611
left a comment
There was a problem hiding this comment.
One last remark, then I think we’re ready to ship.
Apart from that, looking great! You clearly have a much deeper understanding of the QCOW2 spec than I do - your comments are very helpful, and this is definitely heading in the right direction. Thanks!
| if is_writable { | ||
| return Err(Error::CorruptImage); | ||
| } | ||
| warn!("QCOW2 image is marked corrupt, opening read-only"); |
There was a problem hiding this comment.
nit: in case you have that information here
warn!("QCOW2 image is marked corrupt, opening read-only: {file_path}");
There was a problem hiding this comment.
Done by reading the filename from the procfs and falling back to unknown otherwise. Cleaner would be to add a field in RawFile, but that would require API changes at several places which would be imo non worth it. Also, looked up this /proc metod that is used elswhere in the codebase already. And, this will be also helpful in case a corruption concerns a backing file, too. Thanks
| @@ -1499,6 +1527,13 @@ impl QcowFile { | |||
| (address / self.raw_file.cluster_size()) % self.l2_entries | |||
| } | |||
|
|
|||
| // Set corrupt bit without propagating errors. | |||
There was a problem hiding this comment.
Please add 1-2 sentences describing the important why part.
Otherwise, readers may ask:
- why is this called best effort?
- why are we not propagating errors here but everywhere else?
There was a problem hiding this comment.
Indeed, an exhaustive comment in code will stay. Done that. Thanks
Implement proper handling of the QCOW2 corrupt bit (incompatible feature bit 1) according to the specification: - Add Error::CorruptImage for rejecting writable opens of corrupt images - Add CORRUPT to SUPPORTED features (handled specially, not rejected) - Add QcowHeader::set_corrupt_bit() to mark images as corrupt - Add QcowHeader::is_corrupt() helper method - Reject writable opens of corrupt images with Error::CorruptImage - Allow readonly opens of corrupt images with a warning The corrupt bit indicates that image metadata may be inconsistent. Per spec, such images must not be written to until repaired by external tools like qemu-img. Read-only access is permitted to allow data recovery. Users can open corrupt images read-only using: --disk path=/path/to/image.qcow2,readonly=on Signed-off-by: Anatol Belski <[email protected]>
Add comprehensive tests for the corrupt bit handling. Cover writable rejection, read-only access, persistence, and dirty bit coexistence. Signed-off-by: Anatol Belski <[email protected]>
af8c074 to
f029176
Compare
| @@ -1499,6 +1529,19 @@ impl QcowFile { | |||
| (address / self.raw_file.cluster_size()) % self.l2_entries | |||
| } | |||
|
|
|||
| // Attempts to set the corrupt bit, logging failures without propagating them. | |||
There was a problem hiding this comment.
Please use /// - only this way, rustdoc and IDEs with support for that will display the documentation correctly
| // Attempts to set the corrupt bit, logging failures without propagating them. | |
| /// Attempts to set the corrupt bit, logging failures without propagating them. |
I appreciate the new comment, tho!
There was a problem hiding this comment.
Done. Was initially wondering about this since it's a private method and noticed the codebase uses // for most private helpers in this file. But /// does make sense for IDE hover support regardless of visibility. Thanks!
Thanks for the kind words and thorough reviews! Learning ever more while going forward and appreciate the questions, as for answering them I have to verify with the specs which helps to learn more, too :) QCOW is a key format for the OSS virt world and so the proper support is important. Hopefully we can achieve an equivalent state and knowledge to keep on par with QEMU. Thanks! |
Set the QCOW2 corrupt bit when internal inconsistencies are detected that indicate image metadata may be corrupted: - Decompression decode failure, meaning compressed cluster data is invalid - Decompression size mismatch, where decompressed data doesn't match expected cluster size - Partial write after decompression, where L2 table was updated but data cluster not fully written, leaving metadata inconsistent - Invalid refcount index, where cluster address is outside valid refcount table range, indicating a corrupted L2 entry - Dirty L2 with zero L1 address, where L2 table is marked dirty but L1 has no address for it Note: Marking decompression failures as corrupt is more conservative than QEMU, which returns EIO without setting the corrupt bit. This is debatable since corrupted compressed data doesn't necessarily indicate metadata corruption, but it provides a stronger safety guarantee by preventing further writes to potentially damaged images. Once set, the image can only be opened read-only until repaired with qemu-img check -r. Signed-off-by: Anatol Belski <[email protected]>
Validate that L2 table offsets and refcount block offsets are cluster aligned. Set the corrupt bit when unaligned offsets are detected, as this indicates corrupted L1 or refcount table entries. Validate that data cluster offsets from L2 entries are cluster aligned during both reads and writes to existing clusters. Set the corrupt bit when unaligned data cluster offsets are detected. Prevent allocation of clusters at offset 0, which contains the QCOW2 header and should never be allocated. This catches corruption in the available clusters list. Set the corrupt bit when this condition is detected. Signed-off-by: Anatol Belski <[email protected]>
Add integration tests for QCOW2 corrupt bit handling. Verify that images with the corrupt bit set are rejected for writable access but allowed for read-only access with a warning. Helper functions are added to read and modify the corrupt flag in the QCOW2 v3 header. Signed-off-by: Anatol Belski <[email protected]>
f029176 to
07c91a3
Compare
This PR implements support for the QCOW2 corrupt bit (incompatible_features bit 1). The corrupt bit signals that the image is corrupt and must not be modified.
Behavior
Detection triggers
This functionality in geeral matches QEMU's behavior for handling corrupt images.
Differences from QEMU
QEMU provides repair functionality via
qemu-img check -rwhich can fix refcounts, rebuild tables, and reclaim leaked clusters. This implementation only detects and marks corruption but does not include repair capabilities. Corrupt images should be repaired using QEMU tools before use.