Codestin Search App

weltling · 2025-12-17T10:58:27Z

This patch series implements support for RAW backing files in QCOW2 images, enabling QCOW2 overlays to use RAW images as their backing store.

Add RAW format backing file support alongside existing QCOW2 backing support
Implement QCOW2 v3 header extension parsing to read backing file format specifications
For v2 and below, add format auto-detection when backing format is not explicitly specified
Add corresponding functional and performance tests along with some unit test fixes

Performance tests comparing QCOW2 vs RAW backing files show RAW backing provides 43% better sequential read performance (874 vs. 1254 MiB/s) and 5% better random read performance (1625 vs. 1704 MiB/s). Future optimization opportunities remain.

liuw · 2025-12-18T23:04:04Z

We put in the qemu-img check in the test suite, but does it check the backing file's consistency? If not, we should find a way to check that, too.

weltling · 2025-12-19T10:26:40Z

We put in the qemu-img check in the test suite, but does it check the backing file's consistency? If not, we should find a way to check that, too.

Yep, that's a good point. While the backing file itself is readonly, of course one could add some validation in the tests. Fro the runtime, I've noted this, too, will check if some fast method is possible without impacting the perf much, or maybe as an extra option.

Thanks

weltling · 2025-12-19T12:59:42Z

I've now added now the backing file checksum validaiton to the tests. For qcow additionally also qemu-img check ..., too.

Thanks

rbradford

@weltling It's really cool to see improvements to the QCOW2 code - you're becoming our goto expert on that! Since this is pretty isolated code there is good scope for unit testing (vs other parts of the codebase) so i'd encourage you to include more of that.

rbradford · 2026-01-11T10:17:38Z

+    }
+
+    fn compute_file_checksum(path: &std::path::Path) -> u32 {
+        use std::io::Read;


nit: I think we normally put use statements outside the method.

Fixed, thanks! I've overseen it's already imported through a grouped import at the top.

rbradford · 2026-01-11T10:18:21Z

+        initial_backing_checksum: Option<(std::path::PathBuf, String, u32)>,
+    ) {
+        let path = resolve_disk_path(path_or_image_name);
+


nit: This whitespace line feels unnecessary.

Fixed, thanks.

rbradford · 2026-01-11T10:21:42Z

@@ -244,6 +246,7 @@ pub struct QcowHeader {

    // Post-header entries
    pub backing_file_path: Option<String>,
+    pub backing_file_format: Option<String>,


What's that rationale for this being a String and not an enum type?

Good point! While QCOW2 stores it as a string in the header extension, it's effectively a fixed set of formats (raw/qcow2). I've updated it to use ImageType enum with FromStr/Display traits. Thanks!

rbradford · 2026-01-11T10:23:19Z

@@ -319,7 +319,7 @@ mod adjuster {
    }
 }

-const TEST_LIST: [PerformanceTest; 32] = [
+const TEST_LIST: [PerformanceTest; 34] = [


@likebreath Needs to review this file.

Hey @likebreath, would you have time to check this part please? Thanks

rbradford · 2026-01-11T10:24:22Z

@@ -250,6 +255,45 @@ pub struct QcowHeader {
 }

 impl QcowHeader {
+    fn read_header_extensions(f: &mut RawFile, header: &mut QcowHeader) -> Result<()> {


Ideally this function would be unit tested.

ACK. I've added some unit tests specifically for this function, which can be extended once more extensions are parsed. Thanks

rbradford · 2026-01-11T10:28:43Z

+        }
+    }
+
+    fn compute_file_checksum(path: &std::path::Path) -> u32 {


This should probably take & std::io::Read and then be unit tested.

Done. Added unit tests for deterministic hashing and edge cases. Since this is test infrastructure, the unit tests provide extra confidence. Thanks

weltling · 2026-01-11T22:08:51Z

@weltling It's really cool to see improvements to the QCOW2 code - you're becoming our goto expert on that! Since this is pretty isolated code there is good scope for unit testing (vs other parts of the codebase) so i'd encourage you to include more of that.

Thanks! Yep, there are definitely more tests to come and I've already some improvement ideas based on the previous PR conversations. For QCOW2 itself, as the perf validation baseline is being established, I'd anticipate more to achieve especially for the RAW backing file. But also, with #7560 there is yet another wave of improvements to come for the whole ./block module.

phip1611

Generally LGTM, thanks for working on this! I left one remark and a few nits.

phip1611 · 2026-01-12T09:16:16Z

@@ -244,6 +268,7 @@ pub struct QcowHeader {

    // Post-header entries
    pub backing_file_path: Option<String>,
+    pub backing_file_format: Option<ImageType>,


Would it make sense to introduce a new type BackingFileConfig or so type that bundles these properties? Otherwise, one has to check in code for situations where backing_file_format is Some and backing_file_path is None (or vice versa) which to my understanding is invalid anyway.

Yeah, there can be no backing file path without the format and vice versa. Implemented as per your suggestion. Thanks!

phip1611 · 2026-01-12T09:21:19Z

+
+    fn compute_file_checksum(reader: &mut dyn std::io::Read, size: u64) -> u32 {
+        // Read first 16MB or entire data if smaller
+        let read_size = std::cmp::min(size, 16 * 1024 * 1024) as usize;


nit, personal style preference, feel free to ignore: std::cmp::min -> cmp::min + use std::cmp

Fixed, thanks.

phip1611 · 2026-01-12T09:24:16Z


        assert!(
            output.status.success(),
            "qemu-img check failed: {}",
            String::from_utf8_lossy(&output.stderr)
        );
+
+        if let Some((backing_path, format, initial_checksum)) = initial_backing_checksum {
+            if format != "raw" {


I am somewhat concerned that this relies on a "magic value" rather than a well-defined type, constant, or enum variant. However, I am not that familiar with this part of the codebase to judge whether this can be improved easily as a drive-by change.

The format string comes from qemu-img output, that's how it is stored in the image header. But this brought me to another idea - I've added block to the dev dependencies and switched to comparing using block::qcow::ImageType. That's of course much cleaner and indirectly tests the block crate. Otherwise, this condition check is uncritical as it's only about deciding in the test code whether qemu-img check .. is to be run. Thanks!

love it, awesome!

Signed-off-by: Anatol Belski <[email protected]>

phip1611 · 2026-01-12T15:58:17Z

@weltling Since the CI is still failing, here is a small hint: we have invested significant effort into improving developer productivity (#7489). You can just run cargo clippy locally, which should now work (this was not the case prior to v50).

weltling · 2026-01-12T16:04:46Z

@weltling Since the CI is still failing, here is a small hint: we have invested significant effort into improving developer productivity (#7489). You can just run cargo clippy locally, which should now work (this was not the case prior to v50).

Thanks for pointing that out, @phip1611. I indeed looked up the clippy cmd from a failing job. And previously the other one about the bisectability. Got a little bit complicated this round, as many changes were to re-integrate into the history while keeping the history itself untouched, so caught some issues due to the cmoplexity. But should be all fixed now and good to know for the future more things work locally now.

Thanks!

phip1611 · 2026-01-12T19:24:34Z

@weltling Since the CI is still failing, here is a small hint: we have invested significant effort into improving developer productivity (#7489). You can just run cargo clippy locally, which should now work (this was not the case prior to v50).

Thanks for pointing that out, @phip1611. I indeed looked up the clippy cmd from a failing job. And previously the other one about the bisectability. Got a little bit complicated this round, as many changes were to re-integrate into the history while keeping the history itself untouched, so caught some issues due to the cmoplexity. But should be all fixed now and good to know for the future more things work locally now.

Thanks!

I recently learned about git rebase -i HEAD~10 --exec "cargo check && cargo +nightly fmt && cargo clippy". Game changer! It is so easy to style check every single commit and easily amend the corresponding changes

(there might be minor typos in the command as I'm on the phone)

weltling · 2026-01-15T08:47:41Z

@weltling Since the CI is still failing, here is a small hint: we have invested significant effort into improving developer productivity (#7489). You can just run cargo clippy locally, which should now work (this was not the case prior to v50).

Thanks for pointing that out, @phip1611. I indeed looked up the clippy cmd from a failing job. And previously the other one about the bisectability. Got a little bit complicated this round, as many changes were to re-integrate into the history while keeping the history itself untouched, so caught some issues due to the cmoplexity. But should be all fixed now and good to know for the future more things work locally now.
Thanks!

I recently learned about git rebase -i HEAD~10 --exec "cargo check && cargo +nightly fmt && cargo clippy". Game changer! It is so easy to style check every single commit and easily amend the corresponding changes

(there might be minor typos in the command as I'm on the phone)

So far very useful, thanks for the hint. The only pity is it doesn't work in the config rebase.exec. Thanks

phip1611

LGTM! Thanks for your work on this!

Add support for raw backing files in addition to qcow2 backing files. This enables QCOW2 overlays to use raw images as their backing store. The backing file format is auto-detected when not specified, using the existing detect_image_type() function. Add backing_file_format field to QcowHeader to store the format type, which will be populated from header extensions by a subsequent patch. Modify new_from_backing() to accept a backing_format parameter, consolidating support for both raw and qcow2 backing files in a single function. The backing_file_size parameter allows overlay creation without opening the backing file multiple times. Signed-off-by: Anatol Belski <[email protected]>

Add sequential and random read tests for QCOW2 overlays with RAW backing files. Signed-off-by: Anatol Belski <[email protected]>

Add support for parsing QCOW v3 header extensions to read the backing file format. The QCOW v3 spec allows optional header extensions between the fixed header and the backing file name. Implement read_header_extensions() to parse the extension area, which starts at the header_size offset. At the moment it is used to read the backing file format. Further extension processing is open in folow up implementations. Signed-off-by: Anatol Belski <[email protected]>

The write_to() function is used by test code to create qcow2 files for testing. For v3 headers with extended header_size (>104), it needs to: 1. Write the mandatory compression_type field at bytes 104-111 2. Write the header extension end marker at the header_size offset 3. Seek to backing_file_offset before writing the backing file path Additionally, create_for_size_and_path() must set backing_file_offset to account for the 8 byte extension end marker in v3 files, so the backing file path doesn't overwrite the extension area. Add unit tests for read_header_extensions() covering backing format parsing (raw/qcow2), unknown extensions, and error cases (invalid formats, invalid UTF-8). These tests depend on the header writing fixes to create properly formatted v3 test files. Signed-off-by: Anatol Belski <[email protected]>

- Automatically detect and verify backing files - Verify backing file integrity with qemu-img check (qcow only) - Compute DJB2 checksums before test to detect modifications Signed-off-by: Anatol Belski <[email protected]>

…W2 images Backport patches for RAW backing file cloud-hypervisor#7570 - Add RAW format backing file support alongside existing QCOW2 backing support - Implement QCOW2 v3 header extension parsing to read backing file format specifications - For v2 and below, add format auto-detection when backing format is not explicitly specified - Add corresponding functional tests along with some unit test fixes Performance tests are not backported in this PR due to conflicts. They might be considered as a separate effort. Fixes: #60389710 ---- #### AI description (iteration 1) #### PR Classification This PR implements a new feature by adding RAW backing file support for QCOW2 images. #### PR Summary The pull request updates both the image handling logic and associated integration tests to support RAW backing files in QCOW2 images, ensuring proper file integrity and header extension parsing. - **`tests/integration.rs`**: Added functions for computing file checksums, integrated RAW backing file verification into disk consistency checks, and introduced a new test for QCOW2 images with RAW backing files. - **`block/src/qcow/mod.rs`**: Refactored backing file management by introducing a new `BackingFileConfig` structure and `BackingFileOps` trait, and updated header extension parsing to support RAW formats. - **`scripts/run_integration_tests_aarch64.sh` & `scripts/run_integration_tests_x86_64.sh`**: Modified to generate QCOW2 images using a RAW backing file via updated `qemu-img` commands. - **`Cargo.toml` & `Cargo.lock`**: Updated dependencies to include the `block` crate.  Related work items: #60389710

weltling requested a review from a team as a code owner December 17, 2025 10:58

weltling force-pushed the qcow-fixes-backing-raw branch from e69809b to 9f9916c Compare December 17, 2025 11:31

russell-islam reviewed Dec 19, 2025

View reviewed changes

Comment thread block/src/qcow/mod.rs

Comment thread block/src/qcow/mod.rs

Comment thread block/src/qcow/mod.rs

Comment thread block/src/qcow/mod.rs Outdated

weltling force-pushed the qcow-fixes-backing-raw branch 5 times, most recently from b029146 to bc02239 Compare December 20, 2025 08:41

rbradford changed the title ~~bock: qcow: Add RAW backing file support for QCOW2 images~~ block: qcow: Add RAW backing file support for QCOW2 images Dec 23, 2025

russell-islam reviewed Jan 7, 2026

View reviewed changes

Comment thread block/src/qcow/mod.rs

weltling force-pushed the qcow-fixes-backing-raw branch 3 times, most recently from 947a71e to cb8525c Compare January 9, 2026 10:08

rbradford reviewed Jan 11, 2026

View reviewed changes

weltling force-pushed the qcow-fixes-backing-raw branch 2 times, most recently from a665a16 to ca33bf8 Compare January 11, 2026 19:25

phip1611 requested changes Jan 12, 2026

View reviewed changes

weltling force-pushed the qcow-fixes-backing-raw branch from ca33bf8 to ef3e9a1 Compare January 12, 2026 14:38

tests: qcow: Add testing for backing file in raw format

7c26bc5

Signed-off-by: Anatol Belski <[email protected]>

weltling force-pushed the qcow-fixes-backing-raw branch 3 times, most recently from 7d18f95 to 1c16e30 Compare January 12, 2026 15:49

weltling force-pushed the qcow-fixes-backing-raw branch from 1c16e30 to 543b7c3 Compare January 12, 2026 16:00

weltling force-pushed the qcow-fixes-backing-raw branch from 543b7c3 to 3f87b17 Compare January 12, 2026 16:58

weltling requested a review from phip1611 January 14, 2026 08:14

phip1611 approved these changes Jan 15, 2026

View reviewed changes

Comment thread block/src/qcow/mod.rs

Comment thread cloud-hypervisor/tests/integration.rs

Comment thread cloud-hypervisor/tests/integration.rs Outdated

weltling added 4 commits January 15, 2026 15:29

performance-metrics: Add RAW backing file performance tests

9f371f0

Add sequential and random read tests for QCOW2 overlays with RAW backing files. Signed-off-by: Anatol Belski <[email protected]>

weltling force-pushed the qcow-fixes-backing-raw branch 2 times, most recently from b545372 to 8a17944 Compare January 15, 2026 14:32

tests: qcow: Add backing file verification for qcow2 images

6dfb337

- Automatically detect and verify backing files - Verify backing file integrity with qemu-img check (qcow only) - Compute DJB2 checksums before test to detect modifications Signed-off-by: Anatol Belski <[email protected]>

weltling force-pushed the qcow-fixes-backing-raw branch from 8a17944 to 6dfb337 Compare January 15, 2026 15:27

rbradford enabled auto-merge January 15, 2026 16:14

rbradford added this pull request to the merge queue Jan 15, 2026

Merged via the queue into cloud-hypervisor:main with commit 4f69cf6 Jan 15, 2026
45 checks passed

weltling deleted the qcow-fixes-backing-raw branch January 16, 2026 09:20

likebreath added this to Cloud Hypervisor Roadmap Feb 19, 2026

github-project-automation Bot moved this to 🆕 New in Cloud Hypervisor Roadmap Feb 19, 2026

github-project-automation Bot moved this from 🆕 New to ✅ Done in Cloud Hypervisor Roadmap Feb 19, 2026

Uh oh!

Conversation

weltling commented Dec 17, 2025

Uh oh!

liuw commented Dec 18, 2025

Uh oh!

weltling commented Dec 19, 2025

Uh oh!

weltling commented Dec 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weltling commented Jan 11, 2026

Uh oh!

phip1611 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phip1611 commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weltling commented Jan 12, 2026

Uh oh!

phip1611 commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weltling commented Jan 15, 2026

Uh oh!

phip1611 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phip1611 commented Jan 12, 2026 •

edited

Loading

phip1611 commented Jan 12, 2026 •

edited

Loading