Request for discussion: Using PE libraries for Wine dependencies

[Vincent Povirk (they/them)]

The idea of using PE libraries for some Wine library dependencies came
up recently because of a bug in CrossOver's 32-bit support on Mac
(which uses 64-bit system libraries). Wine's xaudio libraries don't do
any translation of 32-bit structures to 64-bit for the native FAudio,
meaning that in this configuration, they don't work at all. If FAudio
and xaudio were both built as a PE, 32-bit binary, we wouldn't need
this translation.

Not all dependencies could be used in PE form, anything that needs to
interact with the host system wouldn't work. We'll always need native
libraries for X11, OpenGL, Vulkan, and host audio libraries, for
example. Only libraries like libpng or FAudio which can function based
on win32 API's we provide (and/or other libraries) could work.

Advantages:
 * This reduces the surface area of Wine's interface between the
virtual Windows environment and host libraries. That makes the
transition to PE simpler, and should make it easier to implement
32-bit with 64-bit host libraries in the official Wine release
(because fewer thunks to 64-bit host libraries will be required).
 * It reduces dependencies on host libraries. If FAudio.dll is shipped
with Wine or in some other way, users won't need an FAudio package
from the host distribution.
 * It allows code sharing between Wine itself and the Mono and Gecko
addons. It's kinda silly that Wine already ships an FAudio.dll, inside
Wine Mono, for its XNA support. Wine's xaudio could theoretically be
sharing this code, but it isn't.

Requirements:
 * Any solution should function even if mingw is not available in the
build environment. This is a challenge because the headers may not
work correctly in the winelib build environment. In particular, the
calling convention is different between 64-bit Windows and everywhere
else, which requires us to decorate Gecko and Mono exports with CDECL
so we can call them correctly. It may be possible to work around this
problem by wrapping the headers with pragma GCC target.
 * We probably shouldn't break anything that works in any currently
supported ports. With llvm-mingw, we now have the ability to build PE
libraries for arm and aarch64 (I've built SDL2 and FAudio this way
myself, but I don't have a test environment for them). I'm not sure of
the status of the PowerPC and sparc ports. Are these maintained? Do we
care? If we do, we'll need to preserve the ability to use native
libraries, or build the libraries in winelib.
 * Our versions of PE dependency libraries must be independent of an
application shipping the same libraries. If we ship an FAudio.dll, and
an application ships an incompatible FAudio.dll, Wine and the
application should each load their own version. This is not an
unlikely scenario, as many libraries make use of C standard library
features such as FILE or malloc/free in their API's, which means that
the ABI depends on which version of the Microsoft C runtime was used
to compile it.

I started a proof of concept, but I got stuck on the question of how
to distribute the FAudio headers, link-time libraries, and binaries.

Part of my proof of concept is a repo on github:
https://github.com/madewokherd/wine-pedeps

Currently, that project builds a couple of PE libraries and does
nothing else. It could be developed further to generate packages of
headers, libraries, and/or binaries. I'll be referring to that project
as wine-pedeps.

Options for distributing dependencies:
 * Distribute headers and libs in a tarball built from wine-pedeps.
The binaries would be part of an addon, also built from wine-pedeps.
 * Import the headers into the Wine tree, and link using LoadLibrary
so we don't need .lib files. The headers could be an output from
wine-pedeps that gets dropped into a folder like
include/wine/external. For building Gecko and Mono, we'd still get
headers and libs from wine-pedeps. No one seems to like this idea.
 * Depend on headers from the host distribution for the corresponding
Linux library, and link using LoadLibrary. I'm not sure if this can
work reliably, or even if we can make host headers accessible inside
mingw without breaking things. For building Gecko and Mono, we'd still
get headers and libs from wine-pedeps.
 * Import entire projects into the Wine tree, perhaps as a submodule.
We would then build them and distribute them with Wine. In case mingw
is not available, we'd need the option of building them with winelib,
or using a host library instead. I don't think the Wine maintainer
likes submodules, and I personally don't like the idea of bloating the
Wine source distribution the way Mono's use of this approach has
forced bloat on the Wine Mono source tree.
 * Rely on the host distribution to package mingw headers, libs, and
binaries for the dependencies we need. The binaries would then be
distributed with Wine. Fedora is the only distribution I'm aware of
that has an extensive library of mingw packages. To make this easier
on other distros, I could maintain wine-pedeps as an alternative
source of headers, libraries, and binaries at build time. The trouble
with this approach is that it's not clear how we can keep Wine's
libraries independent from the application. Perhaps an automated
process could rename the binaries and edit their import tables, or sxs
could be added after the fact.