[RFC PATCH] Improved fake DLLs generation.

[Rémi Bernon]

Hi everyone!

I'm currently trying to make some Call of Duty games work, and for that
I have been working on improving the fake DLLs code, based on the Wine
Staging patch series, but also with a bit of rework.

It's now apparently working well, and I'm sending this here to get some
feedback, as I'm not completely sure of all the implications.

This series addresses several issues at the same time, but the main
idea is to make it possible to use fake DLLs directly from disk, as long
as there's the corresponding builtin so DLL already in memory, and to
make them closely match the module exposed by the builtin DLLs.

The games are doing that for ntdll, and the current PE conversion will
make this superfluous, but CoD Black Ops 3 also does too for gdi32.dll,
calling GdiDllInitialize from a file mapping for instance (and possibly
others I haven't seen yet).

So, this does the following:

* First of all, the NT header generation for the builtin so DLLs is
   improved, to reduce the amount of fixup required when mapping them.
   This is a bit convoluted and it's been split in many patches, but with
   the whole series, the only fixups required are the RVAs if the headers
   end up being unaligned. It could also be simplified by using linker
   scripts, but it also looked more complicated to maintain.

* Then, instead of generating fake module separately from the builtin
   headers, they are extracted directly from the so file, with the PE
   sections memory, after it has been linked. This makes entry points
   aligned between fake DLLs and in memory module. The main problem is
   the module header alignment, that may need fixups, but it can be
   tweaked with base address adjustment if needed.

* Next is the issue when entry points are called directly from the file
   image. I implemented that by reusing on some of the Wine Staging code,
   with a global callback located in TEB Spare2 field (but it could be
   also put somewhere else).

   Every entry point is redirected with linker symbol wrapping (using
   -Wl,--wrap), to some hand written thunks, and two function pointer
   table. One of the table is initially empty, and the other one is
   pre-filled with the real entry point symbols. When the so builtin
   is initially mapped, the first table is filled with the second one,
   and when the thunks are called, they jump directly into it.

   When called from a file mapping, the thunk find the empty function
   table, and then call the global callback, which fill the table from
   the real table of the corresponding loaded builtin so DLL in memory.

* I also reused the staging syscall thunks, but using the symbol
   wrapping, which makes sure that any internal call also goes through
   the thunks -and potential hooks- like it was done for instance for
   NtOpenFile. It's actually only working for cross-source calls, because
   of the limitation of -Wl,--wrap, but it seems to be enough. The same
   mechanism applies to the normal entry point thunks.

It's still a bit a work in progress, and I already know some potential 
issues, and for instance Steam for Windows complains -but not too much- 
about the thunk dll hotpatch prologue being unexpected, but it works
for these games, passing both their module headers checks as well as the 
ntdll and gdi32 calls.

I'm putting everything in one big attachment, as there's many patches 
and to avoid spamming the list, but git am should do the split if you 
want to try.

Cheers,
-- 
Rémi Bernon <rbernon at codeweavers.com>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: RFC-better-fakedlls.patch
Type: text/x-patch
Size: 208315 bytes
Desc: not available
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20200618/cf7a06bf/attachment-0001.bin>