add uc_mem_read_virtual #2121

PhilippTakacs · 2025-02-26T08:46:54Z

New api to read from a vaddr. When using the MMU it's useful to direct read from virtual addresses.

Needs some tests.

I'm not sure about a write_virtual() function. Could also be useful, but is a lot more complex to implement, because of snapshots.

A virtual to physical translate function would be also nice, but currently the tlb_fill() function has no probe argument.

wtdcode · 2025-02-26T08:58:34Z

I have dreamed of this API for quite a while because uc_mem_read/write only deals with physical addresses and super inconvenient with MMU. Thanks for prototyping!

It would be cool to even have translation API as suggested, but many QEMU MMU fill implementation has serious side effects.

Why does "write_vritual" not work with snapshots? What's the current uc_mem_write behavior? Maybe just keep the same behavior.

Lastly please target dev branch instead (for 2.2.0), the master branch is preparing for 2.1.3 and this change can't go into 2.1.3 obviously =).

PhilippTakacs · 2025-02-26T09:12:14Z

Lastly please target dev

Sorry my bad.

Why does "write_vritual" not work with snapshots?

It would work, it's just more code and maybe duplicated code, because of the snapshot handling. With a simple virtual to physical API it would be more simple.

It would be cool to even have translation API as suggested, but many QEMU MMU fill implementation has serious side effects.

In the current QEMU version is tlb_fill() replaced by tlb_fill_align(). This function has a probe parameter, which would be the key for a virtual to physical translation API.

wtdcode · 2025-02-26T10:26:53Z

CI is failing at this moment and I lack time to fix it. Will do later this week.

PhilippTakacs · 2025-02-26T20:21:55Z

CI is failing at this moment

There were also some bugs in my changes which causes the CI to fail.

but many QEMU MMU fill implementation has serious side effects

I have added a testcase (on x86) and read the MMU implementations to verify this would work. As far as I see it works on every MMU expect for SPARC. Would it be OK to just document this issue?

About the write: Would it be OK to just have a read implementation at the moment?

PhilippTakacs · 2025-02-26T20:46:44Z

What's the current uc_mem_write behavior? Maybe just keep the same behavior.

Currently uc_mem_write does COW when memory snapshots are enabled. I would also expect this from a uc_mem_write_virtual function.

The problem is: This is quite complicated code and with the given tlb_vaddr_to_host it would be more complicated to implement for a uc_mem_write_virtual function. Given the fact that I have two times missed some cases in the uc_mem_write function, I would try to avoid implement this without a simple way to translate virtual to physical addresses.

wtdcode · 2025-02-27T02:25:37Z

I just revisited the tlb_vaddr_to_host and noticed the probe parameter, which should avoid the side effects. Sorry I forget the functionality here yesterday without further checking the code. I also saw documents in SPARC and that's okay we can document it and return something like UC_ERR_ARG.

Btw the tlb_fill function indeed has the probe argument, I assume we can expose the translation API directly? Maybe just expose tlb_vaddr_to_host? In this case, we can avoid code duplication by using tlb_vaddr_to_host to convert vaddr to phyaddr and use uc_mem_read/write to do memory operations.

typedef struct CPUClass {
    // ...
    bool (*tlb_fill)(CPUState *cpu, vaddr address, int size,
                     MMUAccessType access_type, int mmu_idx,
                     bool probe, uintptr_t retaddr);
    // ...
}

PhilippTakacs · 2025-03-07T14:56:50Z

So I have added a uc_virtual_to_physical function. The problem I had was: I wanted to (ab)use a function and only add some code to extract the physical address (like I have done with tlb_vaddr_to_host). I have now just added a tlb_vaddr_to_paddr.

I'll try to implement a uc_mem_write_virtual next week.

PhilippTakacs · 2025-03-14T15:08:22Z

The uc_mem_write_virtual function should work, but I haven't had time to test it. I'll write some tests and a bit more documentation. But I have not that much time at the moment.

PhilippTakacs · 2025-04-01T12:19:20Z

@wtdcode write_virtual is also there and tested.

wtdcode · 2025-04-02T10:36:35Z

@wtdcode write_virtual is also there and tested.

Thanks for the brilliant work. I can review only a bit later as I'm having an urgent deadline on April 10.

wtdcode · 2025-04-07T08:22:44Z

uc.c

@@ -773,6 +773,136 @@ static bool check_mem_area(uc_engine *uc, uint64_t address, size_t size)
    return (count == size);
 }

+uc_err uc_virtual_to_physical(uc_engine *uc, uint64_t address, uc_prot prot,


Do we really need prot here? Is there possible that different protections lead to different physical addresses? If the protection is only used for access check, is it trivial to skip such checks and only "translate" addresses for whatever protection? If that is not trivial, we can keep it as it is.

The prot is to have the mmu access rights checked. You might want to check if the corresponding address is write accessible. To check this you can call uc_virtual_to_physical(uc, addr, UC_PROT_WRITE, &paddr) and check the return code.

Also it's necessary to have prot for a potential tlb_fill call.

But for “translation” itself, it has nothing to do with permissions, right? My point is that, from the semantics of the API, protection seems not necessary here though I understand tlb_fill requires this.

For example, assume a memory region located at vaddr 0x2000 with paddr 0x1000 but with PROT_NONE, i.e. no one can access it. But from the perspective of emulator, the mapping is there exactly 0x2000 -> 0x1000 and we should be able to translate addresses no matter the protections.

However, I do understand that qemu code might not design for such cases. Therefore, if you think it is not feasible to translate addresses without protection, we can leave it as it is but with better docs.

wtdcode · 2025-04-07T08:23:12Z

Interrupted by some other things. Will continue my review a bit later.

wtdcode · 2025-04-07T08:30:33Z

uc.c

+}
+
+UNICORN_EXPORT
+uc_err uc_mem_read_virtual(uc_engine *uc, uint64_t address, uc_prot prot,


I just come up with alternative names: uc_vmem_read/write. What do you think?

Sounds good, I'll adopt this.

wtdcode · 2025-04-07T08:31:50Z

qemu/unicorn_common.h

+     * Only page aligned access is allowed,
+     * because tlb_fill() might change the mappings
+     */
+    assert((addr & TARGET_PAGE_MASK) == ((addr + len - 1) & TARGET_PAGE_MASK));


this shall return UC_ERR_ARG or more appropriate errors instead of assertions.

I use asserts for potential unicorn bugs and return errors for user bugs. This internal function is designed to only be called within one page. This is implemented in uc_mem_read_virtual. Returning an error might cause a user to believe it's a bug in there code.

I'll write a comment about this.

Yeah, but addr is from user inputs, no? uc_mem_read_virtual seems using it directly? My concern is that this might cause the assertion too easy to trigger.

Yes addr is (more or less) from user input, but len is enforced to be maximum the rest of the page. What this assert checks is that the full len is on the same page. So the assert is in exactly what uc_mem_read_virtual enforces. When we add other callers, they must handle this internal api the same way.

So I don't see how this assert triggers to early.

wtdcode · 2025-04-07T08:33:28Z

include/unicorn/unicorn.h

+   for detailed error).
+*/
+UNICORN_EXPORT
+uc_err uc_virtual_to_physical(uc_engine *uc, uint64_t address, uint32_t prot,


I would suggest uc_vmem_translate. What do you think?

wtdcode · 2025-04-07T08:35:02Z

include/unicorn/unicorn.h

+
+ @uc:      handle returned by uc_open()
+ @address: starting virtual memory address of bytes to get.
+ @prot:    The access type for the tlb lookup


As reviewed previously, if @prot is essential, please document a bit more here. Like how different protections could affect the behavior. For example, what does uc_mem_read_virtual with prot = UC_PROT_WRITE imply?

New api to read from, translate, or write to a virtual address. When using the MMU it's useful to direct use the virtual addresses.

PhilippTakacs · 2025-04-29T09:28:46Z

@wtdcode is there anything missing for this getting merged?

wtdcode · 2025-04-30T20:47:57Z

@wtdcode is there anything missing for this getting merged?

Hello Philipp, sorry for late because I'm on academic travel. I will give another review shortly.

PhilippTakacs force-pushed the read-virtuall branch from a977b1e to 214c007 Compare February 26, 2025 08:58

PhilippTakacs changed the base branch from master to dev February 26, 2025 09:02

PhilippTakacs force-pushed the read-virtuall branch from 214c007 to 74ef179 Compare February 26, 2025 10:24

PhilippTakacs force-pushed the read-virtuall branch from 7966d1b to 6461b19 Compare March 7, 2025 12:04

PhilippTakacs marked this pull request as ready for review March 7, 2025 12:07

PhilippTakacs force-pushed the read-virtuall branch 2 times, most recently from be29636 to 013877e Compare March 7, 2025 14:48

PhilippTakacs force-pushed the read-virtuall branch from 2900995 to 208eef5 Compare April 1, 2025 11:34

wtdcode reviewed Apr 7, 2025

View reviewed changes

x86 mmu allow probe requests

8e02438

PhilippTakacs force-pushed the read-virtuall branch from 208eef5 to 745eaf0 Compare April 16, 2025 13:09

add api to access memory by virtuall addresses

1267722

New api to read from, translate, or write to a virtual address. When using the MMU it's useful to direct use the virtual addresses.

PhilippTakacs force-pushed the read-virtuall branch from 745eaf0 to 1267722 Compare April 16, 2025 13:15

fixup! add api to access memory by virtuall addresses

65f3962

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add uc_mem_read_virtual #2121

add uc_mem_read_virtual #2121

PhilippTakacs commented Feb 26, 2025

wtdcode commented Feb 26, 2025 •

edited

Loading

PhilippTakacs commented Feb 26, 2025

wtdcode commented Feb 26, 2025

PhilippTakacs commented Feb 26, 2025

PhilippTakacs commented Feb 26, 2025

wtdcode commented Feb 27, 2025 •

edited

Loading

PhilippTakacs commented Mar 7, 2025

PhilippTakacs commented Mar 14, 2025

PhilippTakacs commented Apr 1, 2025

wtdcode commented Apr 2, 2025

wtdcode Apr 7, 2025

PhilippTakacs Apr 7, 2025

wtdcode Apr 7, 2025

wtdcode Apr 7, 2025 •

edited

Loading

wtdcode commented Apr 7, 2025

wtdcode Apr 7, 2025 •

edited

Loading

PhilippTakacs Apr 7, 2025

wtdcode Apr 7, 2025

PhilippTakacs Apr 7, 2025

wtdcode Apr 7, 2025

PhilippTakacs Apr 7, 2025

wtdcode Apr 7, 2025

wtdcode Apr 7, 2025

PhilippTakacs commented Apr 29, 2025

wtdcode commented Apr 30, 2025

add uc_mem_read_virtual #2121

Are you sure you want to change the base?

add uc_mem_read_virtual #2121

Conversation

PhilippTakacs commented Feb 26, 2025

wtdcode commented Feb 26, 2025 • edited Loading

PhilippTakacs commented Feb 26, 2025

wtdcode commented Feb 26, 2025

PhilippTakacs commented Feb 26, 2025

PhilippTakacs commented Feb 26, 2025

wtdcode commented Feb 27, 2025 • edited Loading

PhilippTakacs commented Mar 7, 2025

PhilippTakacs commented Mar 14, 2025

PhilippTakacs commented Apr 1, 2025

wtdcode commented Apr 2, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wtdcode Apr 7, 2025 • edited Loading

Choose a reason for hiding this comment

wtdcode commented Apr 7, 2025

wtdcode Apr 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PhilippTakacs commented Apr 29, 2025

wtdcode commented Apr 30, 2025

wtdcode commented Feb 26, 2025 •

edited

Loading

wtdcode commented Feb 27, 2025 •

edited

Loading

wtdcode Apr 7, 2025 •

edited

Loading

wtdcode Apr 7, 2025 •

edited

Loading