Request for discussion: Using PE libraries for Wine dependencies

Sat Apr 11 18:43:28 CDT 2020

On 2/14/20 7:43 PM, Vincent Povirk (they/them) wrote:
> The idea of using PE libraries for some Wine library dependencies came
> up recently because of a bug in CrossOver's 32-bit support on Mac
> (which uses 64-bit system libraries). Wine's xaudio libraries don't do
> any translation of 32-bit structures to 64-bit for the native FAudio,
> meaning that in this configuration, they don't work at all. If FAudio
> and xaudio were both built as a PE, 32-bit binary, we wouldn't need
> this translation.
> 
> Not all dependencies could be used in PE form, anything that needs to
> interact with the host system wouldn't work. We'll always need native
> libraries for X11, OpenGL, Vulkan, and host audio libraries, for
> example. Only libraries like libpng or FAudio which can function based
> on win32 API's we provide (and/or other libraries) could work.
> 
> Advantages:
>   * This reduces the surface area of Wine's interface between the
> virtual Windows environment and host libraries. That makes the
> transition to PE simpler, and should make it easier to implement
> 32-bit with 64-bit host libraries in the official Wine release
> (because fewer thunks to 64-bit host libraries will be required).
>   * It reduces dependencies on host libraries. If FAudio.dll is shipped
> with Wine or in some other way, users won't need an FAudio package
> from the host distribution.
>   * It allows code sharing between Wine itself and the Mono and Gecko
> addons. It's kinda silly that Wine already ships an FAudio.dll, inside
> Wine Mono, for its XNA support. Wine's xaudio could theoretically be
> sharing this code, but it isn't.
> 
> Requirements:
>   * Any solution should function even if mingw is not available in the
> build environment. This is a challenge because the headers may not
> work correctly in the winelib build environment. In particular, the
> calling convention is different between 64-bit Windows and everywhere
> else, which requires us to decorate Gecko and Mono exports with CDECL
> so we can call them correctly. It may be possible to work around this
> problem by wrapping the headers with pragma GCC target.
>   * We probably shouldn't break anything that works in any currently
> supported ports. With llvm-mingw, we now have the ability to build PE
> libraries for arm and aarch64 (I've built SDL2 and FAudio this way
> myself, but I don't have a test environment for them). I'm not sure of
> the status of the PowerPC and sparc ports. Are these maintained? Do we
> care? If we do, we'll need to preserve the ability to use native
> libraries, or build the libraries in winelib.
>   * Our versions of PE dependency libraries must be independent of an
> application shipping the same libraries. If we ship an FAudio.dll, and
> an application ships an incompatible FAudio.dll, Wine and the
> application should each load their own version. This is not an
> unlikely scenario, as many libraries make use of C standard library
> features such as FILE or malloc/free in their API's, which means that
> the ABI depends on which version of the Microsoft C runtime was used
> to compile it.
> 
> I started a proof of concept, but I got stuck on the question of how
> to distribute the FAudio headers, link-time libraries, and binaries.
> 
> Part of my proof of concept is a repo on github:
> https://github.com/madewokherd/wine-pedeps
> 
> Currently, that project builds a couple of PE libraries and does
> nothing else. It could be developed further to generate packages of
> headers, libraries, and/or binaries. I'll be referring to that project
> as wine-pedeps.
> 
> Options for distributing dependencies:
>   * Distribute headers and libs in a tarball built from wine-pedeps.
> The binaries would be part of an addon, also built from wine-pedeps.
>   * Import the headers into the Wine tree, and link using LoadLibrary
> so we don't need .lib files. The headers could be an output from
> wine-pedeps that gets dropped into a folder like
> include/wine/external. For building Gecko and Mono, we'd still get
> headers and libs from wine-pedeps. No one seems to like this idea.
>   * Depend on headers from the host distribution for the corresponding
> Linux library, and link using LoadLibrary. I'm not sure if this can
> work reliably, or even if we can make host headers accessible inside
> mingw without breaking things. For building Gecko and Mono, we'd still
> get headers and libs from wine-pedeps.
>   * Import entire projects into the Wine tree, perhaps as a submodule.
> We would then build them and distribute them with Wine. In case mingw
> is not available, we'd need the option of building them with winelib,
> or using a host library instead. I don't think the Wine maintainer
> likes submodules, and I personally don't like the idea of bloating the
> Wine source distribution the way Mono's use of this approach has
> forced bloat on the Wine Mono source tree.
>   * Rely on the host distribution to package mingw headers, libs, and
> binaries for the dependencies we need. The binaries would then be
> distributed with Wine. Fedora is the only distribution I'm aware of
> that has an extensive library of mingw packages. To make this easier
> on other distros, I could maintain wine-pedeps as an alternative
> source of headers, libraries, and binaries at build time. The trouble
> with this approach is that it's not clear how we can keep Wine's
> libraries independent from the application. Perhaps an automated
> process could rename the binaries and edit their import tables, or sxs
> could be added after the fact.
> 

What about extending a bit the Wine build system to support external 
sources? Then you could configure Wine with:

   configure --with-sdl2-src=... --with-faudio-src=...

And that would build wine-specific PE wrappers from the sources of the 
projects, and use the headers from there when building the dependent DLLs.

If the sources aren't provided, then fallback to the current 
implementation with system dependencies and ELF builtins instead.

That way we won't have much pollution in Wine's tree, we just need an 
annoying list of sources, a spec file, and maybe a custom config like 
for SDL2. It also makes it easier for anyone to customize the 
dependencies without having to go through a separate build process.

I'm attaching a PoC implementation to illustrate, it's not super clean 
and there's a lot of build warnings but it appears to be working. And 
the xaudio family libraries can then be built as PE.

Now I'm not completely sure how symbol resolution works on Windows, but 
if my understanding is correct there should be no conflicts -or we'd 
also have to rename all the exported functions.

Cheers,
-- 
Rémi Bernon <rbernon at codeweavers.com>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-xaudio-Use-Win32-synchronization-primitives.patch
Type: text/x-patch
Size: 8006 bytes
Desc: not available
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20200412/eb1c394e/attachment-0010.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-xaudio-Build-with-msvcrt-if-possible.patch
Type: text/x-patch
Size: 16171 bytes
Desc: not available
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20200412/eb1c394e/attachment-0011.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-configure-Export-some-variables-for-extern-sources.patch
Type: text/x-patch
Size: 1376 bytes
Desc: not available
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20200412/eb1c394e/attachment-0012.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0004-makedep-Add-EXTERNSRC-variable-for-external-sources.patch
Type: text/x-patch
Size: 1841 bytes
Desc: not available
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20200412/eb1c394e/attachment-0013.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0005-makedep-Pass-make-object-to-other-functions.patch
Type: text/x-patch
Size: 15878 bytes
Desc: not available
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20200412/eb1c394e/attachment-0014.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0006-makedep-Do-not-check-includes-in-external-sources.patch
Type: text/x-patch
Size: 717 bytes
Desc: not available
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20200412/eb1c394e/attachment-0015.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0007-makedep-Allow-relative-includes-in-external-sources.patch
Type: text/x-patch
Size: 834 bytes
Desc: not available
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20200412/eb1c394e/attachment-0016.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0008-makedep-Look-for-includes-in-extern-source-dirs.patch
Type: text/x-patch
Size: 2633 bytes
Desc: not available
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20200412/eb1c394e/attachment-0017.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0009-wine-sdl2-Build-from-external-SDL2-sources.patch
Type: text/x-patch
Size: 9917 bytes
Desc: not available
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20200412/eb1c394e/attachment-0018.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0010-wine-faudio-Build-from-external-FAudio-sources.patch
Type: text/x-patch
Size: 8742 bytes
Desc: not available
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20200412/eb1c394e/attachment-0019.bin>