Access to graphics memory mappings on upcoming WOW64 implementation

[Chris Robinson]

On Sunday, April 24, 2022 7:18:49 PM PDT Derek Lesho wrote:
> The solutions which I've seen laid out so far:
> 
> - Use the mremap(2) interface, allowing us to duplicate the mapping we
> receive into the 32-bit address space.  This solution would match what
> is already done for Crossover Mac's 32on64 support using Mac's
> mach_vm_remap functionality [2].  However, right now it is not possible
> to use the MREMAP_DONTUNMAP flag with mappings that aren't private and
> anonymous, which rules out there use on mapped FDs from libdrm.  Due to
> this, a kernel change would be necessary.

This doesn't sound at all safe, since it's essentially moving the memory 
mapping out from under the driver. MREMAP_DONTUNMAP avoids unmapping the 
original memory space, but that memory space becomes all but invalid, a page 
fault when accessed and is otherwise zero-filled to satisfy accesses:

    MREMAP_DONTUNMAP (since Linux 5.7)
        ...
        After completion, any access to the range specified by old_address and
        old_size will result in a page fault. The page fault will be handled
        by a userfaultfd(2) handler if the address is in a range previously
        registered with userfaultfd(2). Otherwise, the kernel allocates a
        zero-filled page to handle the fault.

You can't know what a driver will do with mapped memory or pointer addresses 
it returns to the application, or where such memory comes from, so you can't 
be sure it doesn't have some bookkeeping with it or does manual copying using 
a cached pointer instead of the remapped location. You also can't know if it's 
using a preallocated pool that it returns to the app when "mapping" and reuses 
after "unmapping".

What you'd need for something like this is a method to duplicate a memory 
mapping, leaving the original intact instead of wiping it, so different pages/
addresses refer to the same underlying hardware memory. There doesn't seem to 
be an option for that, currently.

> - Work with Khronos to introduce extensions into the relevant APIs
> enabling us to tell drivers where in the address space we want resources
> mapped.

This seems like the only real option to me. It's the only way to be sure a 
driver knows what you actually want and won't break some assumptions when it's 
memory mappings are changed on it. This can also be useful in other non-Wine 
situations where a 64-bit app is running native 32-bit code and needs GPU 
memory in the lower 32-bit address space. It can even tell the driver to keep 
unmappable memory (memory the app itself won't directly access) out of the 32-
bit address space to leave more for the 32-bit code, where in a pure 64-bit 
process such a thing wouldn't matter and it may not bother to.

> - Hook the driver's mmap call when we invoke memory mappings function,
> overriding the address to something in the 32-bit address space.

Similar to point 1, you can't be sure how the driver handles memory mapping. 
It could have preallocated memory that mapping simply returns a chunk of, 
meaning there wouldn't be an mmap call during the mapping function since it 
was done some time earlier. On 64-bit systems, the driver could also use a 
memory management style that's more efficient with a large address space 
instead of a smaller one. If you simply force 32-bit addresses on the driver, 
it could make the driver's memory management less efficient or be more 
wasteful with the already-limited 32-bit address space. Explicitly telling the 
driver you want 32-bit addresses for mapped memory would ensure the driver 
knows it needs to be more frugal with mappable memory.