Enable AArch64 #1126

michael2012z · 2020-05-08T05:33:35Z

This pull request contains the code to enable the basic functionalities of Cloud-hypervisor on AArch64.

Summary of the major changes:

Enabled vm-allocator, device manager, memory manager and vcpu manager for AArch64.
Enabled Virtio devices with MMIO transport option.
Configured VM: load kernel, setup GIC setup and generate FDT.
Exit VMM when guest shutdown on AArch64.
Added a guide for testing on Arm64.

What was tested:

Booting a VM (Ubuntu 18.04) with up to 4 VCPUs.
Booting a VM with up to 256 GiB memory.
Serial I/O.
Virtio devices with MMIO transport option: virtio-blk and virtio-console devices were tested.

rbradford · 2020-05-08T09:06:55Z

@michael2012z Thanks for your contribution. This is a huge and significant contribution for your first one!

I can see that you have composed the PR of many well structured commits but we may ask you to revisit the way aspects of this PR are structured (possibly even splitting it if necessary.)

* Now Cloud-hypervisor forked these 2 crates and made necessary modification for X86, but they havenot been ready on ARM. The temporary workaround is modifying some Cargo.toml files to use offical versions of kvm-bindings and kvm-ioctls (without serde support) before building on ARM.

In your PR you have the lines that change the change the dependencies commented out?

For kvm-bindings our ch branch has everything that upstream master has (plus extras) are you saying that you have to use the older 0.2.0 tag because master does not work

For kvm-ioctls I can see we have diverged slightly and we should be able to rebase the changes on top.

michael2012z · 2020-05-08T11:54:47Z

Hi, @rbradford

When I tried to build on an ARM machine, some errors were seen:

   Compiling kvm-bindings v0.2.0 (https://github.com/cloud-hypervisor/kvm-bindings?branch=ch#3a678008)
error: cannot find derive macro `Deserialize` in this scope
  --> /usr/local/rust/git/checkouts/kvm-bindings-c6b55506be61a061/3a67800/src/arm64/mod.rs:29:43
   |
29 | #[cfg_attr(feature = "with-serde", derive(Deserialize, Serialize))]
   |                                           ^^^^^^^^^^^

error: cannot find derive macro `Serialize` in this scope
  --> /usr/local/rust/git/checkouts/kvm-bindings-c6b55506be61a061/3a67800/src/arm64/mod.rs:29:56
   |
29 | #[cfg_attr(feature = "with-serde", derive(Deserialize, Serialize))]
   |                                                        ^^^^^^^^^

ch branch of kvm-bindings has some modifications for Serde, but maybe it's not ready for AArch64. So I have to fallback to the upstream version without Serde to workaround the error. The modification of kvm-ioctls dependency shared similar reason.

And I kept my local modification as comments to remind, if someones want to try on ARM.

michael2012z · 2020-05-08T12:00:12Z

I can see that you have composed the PR of many well structured commits but we may ask you to revisit the way aspects of this PR are structured (possibly even splitting it if necessary.)

Do you suggest to split this single PR into several smaller ones?

rbradford · 2020-05-08T14:51:11Z

Do you suggest to split this single PR into several smaller ones?

We'll let you know how we'd like to see it split up but don't do any work on this at the moment.

michael2012z · 2020-05-08T23:03:07Z

Thanks. I will wait for your advice and do the restructuring then.

I am wondering why the CI jobs hang unfinished. I saw it also happened sometimes in other PR's.

rbradford · 2020-05-11T15:03:42Z

@michael2012z Again thank you for submitting this PR. Please could you extract the following changes into separate PRs so that we can start making progress on merging this.

Firstly can you separate the abstraction and refactoring you did for the IOAPIC (the InterruptController trait etc.)

Secondly can you create the most minimal PR that makes the code build (I don't expect it to be functional) with the rust stable aarch64-unknown-linux-musl toolchain. Adding it to the GitHub actions so that we can build test the code.

Longer term we will need to establish a CI to be able to try and run the integration test suite.

With those two PRs reviewed and merged I think the PR will be a lot less intrusive and more easily reviewable. It's also possible that some of the other "portability" fixes would also benefit from being in their own PR.

Thanks, I look forward to seeing your submissions.

michael2012z · 2020-05-12T02:40:52Z

Hi, @rbradford

Nice to see your feedback. Besides the splitting of InterruptController and porting code you mentioned, I am also going to separate more from this PR. The plan is:

Split the update on Dockerfile & build scripts in a separate PR. And fix all building errors on ARM. Just to confirm, do you mean the same thing by can you create the most minimal PR that makes the code build?
Separate PR for arch/, including layout design and porting code of GIC, FDT.
Separate PR for RTC (porting code). Several files were included, better to put them into a PR.
Refactoring IOAPIC.
Refactoring booting sequence. A big change was made in booting a VM, especially in creating and activating VCPU's. Better to use a separate PR to review and merge.
Other things (memory manager, vm allocator, secomp, etc.) will remain in current PR to finally enable AArch64.

rbradford · 2020-05-12T09:44:35Z

Just to confirm, do you mean the same thing by can you create the most minimal PR that makes the code build?

Using the cross toolchain the compilation succeeds (when started by a GitHub action.)

egernst · 2020-05-12T18:30:10Z

Is it feasible to augment existing device support documentation in order to clarify the 'support level' per architecture, as we expand the architecture support?

Nice work @michael2012z

michael2012z · 2020-05-13T10:26:17Z

@egernst Thanks.

Yes, that document should be updated to reflect the difference between architectures.

michael2012z · 2020-06-10T05:58:28Z

Hi, @rbradford, @sboeuf Having much code separated, this PR came back with ARM-enabling code only. Could you help review it?

@egernst Regarding the documents, now I wrote a very small guide for testing on Arm64. More documenting work is not included in this PR. My personal opinion is it's better update device supporting document when we have CI (in progress now) and have more things stably tested.

rbradford

Looking very good. Could you do a quick pass through the commit messages and:

Standardise on aarch64 or arm64 (or whichever :))
Try and make the commit messages describe WHY rather than WHAT (the code should ideally show the what).
I think some of the commit messages have some strange wrapping issues.

As always look for ways to minimise the amount of compile-time differences even if that means refactoring the existing code.

rbradford · 2020-06-10T13:00:53Z

vm-allocator/src/gsi.rs

+    pub fn allocate_irq(&mut self) -> Result<u32> {
+        self.next_irq = self.next_irq.checked_add(1).ok_or(Error::Overflow)?;
+        Ok(self.next_irq - 1)


Purely as a matter of taste I prefer:

let irq = self.next_irq; self.next_irq = self.next_irq.checked_add(1).ok_or(Error::Overflow)?; Ok(irq)

This is better than using an additional subtracting.

rbradford · 2020-06-10T13:02:06Z

vm-allocator/src/gsi.rs


+    #[cfg(target_arch = "x86_64")]
    /// Allocate an IRQ
    pub fn allocate_irq(&mut self) -> Result<u32> {


Not for you to fix but if we only support one IOAPIC (@sboeuf - would we need more?) then maybe we could heavily simplify this function and make it look like the ARM version.

vmm/src/device_manager.rs

rbradford · 2020-06-10T13:07:22Z

vmm/src/device_manager.rs

            ConsoleOutputMode::Off | ConsoleOutputMode::Null => None,
        };
        let serial = if serial_config.mode != ConsoleOutputMode::Off {
+            #[cfg(target_arch = "x86_64")]


Please refactor this into two functions; one for each architecture. I think it will be much easier to read (e.g. add_serial_device(...)

Good idea. Thanks.

rbradford · 2020-06-10T13:08:55Z

vmm/src/device_manager.rs

        };

+        #[cfg(target_arch = "aarch64")]
+        let device_type = virtio_device.lock().unwrap().device_type();


Could you move this line lower and then use block syntax for the architecture control

virtio_device is consumed among next lines, I can't move it down.
I chose to move self.id_to_dev_info.insert() up, and then have to move irq_num in front of them.

vmm/src/cpu.rs

rbradford · 2020-06-10T13:11:55Z

vm-allocator/src/system.rs

-        io_base: GuestAddress,
-        io_size: GuestUsize,
+        #[cfg(target_arch = "x86_64")] io_base: GuestAddress,
+        #[cfg(target_arch = "x86_64")] io_size: GuestUsize,


Rather than all this complexity. What about making it an Option<T> type. You could also do the same for everywhere the io_bus is used. That would remove a bunch of conditionals and make the code easier to read.

My original consideration to screen IO things with conditions is:

IO is not for ARM, having it in binary looks weird.

Hiding IO bus may avoid potential mistakes in code. It should be wrong if some ARM code is using IO operating/bus, but this will be impossible if IO things are not present. I also planed to make PciBarRegionType::IORegion x86_64 only when coming to support PCI.

vmm/src/cpu.rs

vmm/src/vm.rs

Implemented GSI allocator and system allocator for AArch64. Renamed some layout definitions to align more code between architectures. Signed-off-by: Michael Zhao <[email protected]>

Screened IO bus because it is not for AArch64. Enabled Serial, RTC and Virtio devices with MMIO transport option. Signed-off-by: Michael Zhao <[email protected]>

Screened IO space as it is not available on AArch64. Signed-off-by: Michael Zhao <[email protected]>

Added MPIDR which is needed in system configuration. Signed-off-by: Michael Zhao <[email protected]>

Signed-off-by: Michael Zhao <[email protected]>

X86 and AArch64 work in different ways to shutdown a VM. X86 exit VMM event loop through ACPI device; AArch64 need to exit from CPU loop of a SystemEvent. Signed-off-by: Michael Zhao <[email protected]>

The support of AArch64 is in very early stage. The steps in building and runing on X86 and AArch64 can not align well yet. Adding AArch64 content to README.md would produce much divergence. Adding a guide in docs/ folder could be a better way to start now. Signed-off-by: Michael Zhao <[email protected]>

michael2012z · 2020-06-11T08:14:57Z

vmm/src/vm.rs

+            .clone();
+
+        arch::configure_system(
+            &self.memory_manager.lock().as_ref().unwrap().fd,


Echoing comment #1126 (comment)
The fd was added for this. Now I set fd field public in memory manager and BORROW it back from there.

michael2012z · 2020-06-11T08:31:37Z

Thanks for reviewing.
Code was updated.
I also fixed commit messages. Some messages were removed because the titles explain.

rbradford · 2020-06-11T09:52:06Z

@sboeuf Please review

rbradford

I think this looks pretty good to me!

sboeuf · 2020-06-11T13:00:05Z

docs/arm64.md

+
+## Prerequisites
+
+On Arm64 machines, Cloud-hypervisor depends on an external library `libfdt-dev` for generating Flatted Device Tree (FDT).


Typo: Flatted should be Flattened

@michael2012z Can you please fix this in a subsequent PR.

Thanks, @sboeuf , @rbradford . I will fix it in my next PR.

michael2012z changed the title ~~Aarch64 enable~~ Enable AArch64 May 8, 2020

michael2012z mentioned this pull request May 13, 2020

AArch64 initial build #1168

Merged

michael2012z changed the title ~~Enable AArch64~~ [WIP] Enable AArch64 May 14, 2020

MrXinWang mentioned this pull request May 22, 2020

AArch64: Adding AArch64 arch-related files #1223

Merged

This was referenced May 25, 2020

Refactor IOAPIC #1232

Merged

vmm: Change booting process to cover AArch64 requirements #1238

Merged

This was referenced Jun 3, 2020

AArch64 CI Enablement #1263

Closed

device: Add AArch64 RTC PL031 implementation #1293

Merged

michael2012z force-pushed the aarch64_enable branch from fd0ae6a to 24f6686 Compare June 10, 2020 03:38

michael2012z changed the title ~~[WIP] Enable AArch64~~ Enable AArch64 Jun 10, 2020

michael2012z force-pushed the aarch64_enable branch 2 times, most recently from 51bd18d to f1f5825 Compare June 10, 2020 03:59

michael2012z force-pushed the aarch64_enable branch from f1f5825 to 344f976 Compare June 10, 2020 06:26

rbradford reviewed Jun 10, 2020

View reviewed changes

michael2012z added 5 commits June 11, 2020 15:55

vm-allocator: Enable vm-allocator for AArch64

7fefaa6

Implemented GSI allocator and system allocator for AArch64. Renamed some layout definitions to align more code between architectures. Signed-off-by: Michael Zhao <[email protected]>

vmm: Enable device manager for AArch64

e714b28

Screened IO bus because it is not for AArch64. Enabled Serial, RTC and Virtio devices with MMIO transport option. Signed-off-by: Michael Zhao <[email protected]>

vmm: Enable memory manager for AArch64

4a49a51

Screened IO space as it is not available on AArch64. Signed-off-by: Michael Zhao <[email protected]>

vmm: Enable VCPU for AArch64

f4a6734

Added MPIDR which is needed in system configuration. Signed-off-by: Michael Zhao <[email protected]>

vmm: Configure VM on AArch64

30804bf

Signed-off-by: Michael Zhao <[email protected]>

michael2012z added 2 commits June 11, 2020 15:55

vmm: Exit VMM event loop after guest shutdown for AArch64

cee881f

X86 and AArch64 work in different ways to shutdown a VM. X86 exit VMM event loop through ACPI device; AArch64 need to exit from CPU loop of a SystemEvent. Signed-off-by: Michael Zhao <[email protected]>

michael2012z force-pushed the aarch64_enable branch from 344f976 to 1bebeca Compare June 11, 2020 07:56

michael2012z commented Jun 11, 2020

View reviewed changes

rbradford approved these changes Jun 11, 2020

View reviewed changes

sboeuf reviewed Jun 11, 2020

View reviewed changes

sboeuf approved these changes Jun 11, 2020

View reviewed changes

rbradford merged commit 3f18f93 into cloud-hypervisor:master Jun 11, 2020

michael2012z mentioned this pull request Jun 12, 2020

AArch64 support #1125

Closed

15 tasks

MrXinWang mentioned this pull request Jun 12, 2020

unit tests: Fix unit tests and docs for AArch64 #1318

Merged


		## Prerequisites

		On Arm64 machines, Cloud-hypervisor depends on an external library `libfdt-dev` for generating Flatted Device Tree (FDT).

Enable AArch64 #1126

Enable AArch64 #1126

Uh oh!

Conversation

michael2012z commented May 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rbradford commented May 8, 2020

Uh oh!

michael2012z commented May 8, 2020

Uh oh!

michael2012z commented May 8, 2020

Uh oh!

rbradford commented May 8, 2020

Uh oh!

michael2012z commented May 8, 2020

Uh oh!

rbradford commented May 11, 2020

Uh oh!

michael2012z commented May 12, 2020

Uh oh!

rbradford commented May 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

egernst commented May 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michael2012z commented May 13, 2020

Uh oh!

michael2012z commented Jun 10, 2020

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michael2012z Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michael2012z commented Jun 11, 2020

Uh oh!

rbradford commented Jun 11, 2020

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

michael2012z commented May 8, 2020 •

edited

Loading

rbradford commented May 12, 2020 •

edited

Loading

egernst commented May 12, 2020 •

edited

Loading

michael2012z Jun 11, 2020 •

edited

Loading