Request for discussion: Using PE libraries for Wine dependencies

[Jacek Caban]

Hi Zebediah,

I really think we should at least experiment and evaluate using winelib 
first. My previous mail didn't really explain details behind this and 
was a bit messy, so let me try now.

First, the name mingw or mingw-w64 is used to name different things 
depending on the context. For sake of this mail, let me use it to call 
the complete toolchain for (cross, but not only) compiling a Window 
target. mingw based on GCC contains following parts from different 
projects (LLVM mingw variant is similar, except that llvm/clang is 
always a cross compiler, so this part doesn't need to be built with 
mingw in mind and instead a fairly regular build can be used):

- GCC + binutils tools built with mingw as a target (from gcc and 
binutils tree)
- libgcc cross compiled for mingw target (from gcc tree, but depends on 
crt part of mingw-w64)
- crt headers (from mingw-w64 repo)
- crt static libraries and startup files (libmingwex, libmingw32, 
libmsvcrt, etc, from mingw-w64 repo)
- platform headers (from mingw-w64 repo)
- platform importlibs (from mingw-w64 repo)
- optionally winpthreads, which messes with crt headers and crt libs 
(from mingw-w64 repo)
- libstdc++ and a few more component not important for sake of this 
conversation

Wine is not a typical mingw user, because it actually implements the 
platform for which it builds. As such, we need to provide majority of 
the stuff that other projects would use from mingw-w64. Initially, Wine 
used an exotic mix for crt: it used its own headers and msvcrt 
importlib, but mingw-w64 for the rest of static libs and startup files. 
It meant that Wine dependent on mingw-w64 internals that it shouldn't 
touch. mingw-w64 assumes that crt headers + crt libs are shipped 
together and it has the right to do so. It worked, but it was fragile. 
Anything that Wine didn't take into account (like configuring mingw-w64 
to use ucrt by default) could cause a problem. It's probably just a 
matter of time when wine-5.0 will not build with future mingw-w64 
releases and it will not be mingw-w64's fault.

We looked for a solution to that problem. One way to solve this would be 
tightening projects relationships. It's wasn't clear how technically 
that could look, but we had to put that idea aside for non-technical 
reasons anyway.

The other solution is to not mix crt parts at all. This required Wine to 
provide the few missing bits (mostly startup files) by itself. I'd say 
that it makes sense: we provide similar things for ELF builds anyway and 
we provide msvcrt implementation and headers, it feels natural to 
provide static library bits as well. Since February (or August last year 
for majority of the codebase), Wine does not use any part of mingw-w64 
repo directly at all. We always use our own headers and -nodefaultlibs. 
Wine cross toolchain dependency is reduced to just GCC + binutils + 
libgcc (and mingw-w64 as build time libgcc dependency) from the list 
above. That's what you get with winegcc -mno-cygwin (without 
-mno-cygwin, we still use the exotic mix for mingw targets and we still 
use that for PE-only builds).

So now that we have this powerful tool and successfully use it for Wine 
build, it feels natural and technically right to use it for the rest of 
Wine ecosystem. What you propose would instead bring back mixing crt 
parts. Linking current Wine against external mingw-compiled (as opposed 
to winegcc -mno-cygwin) library is essentially linking a static lib with 
a different ABI. Headers should mostly work, but it's still shaky.

I didn't try winegcc yet with non-Wine code myself mostly because I 
still think there are things that we should get right first. My ucrt 
work was partially motivated by this. We should be able to default to 
ucrt in winegcc now (we already do that for Wine in makedep, so I'd say 
we're ready for that). There are a few other things, but we're getting 
there.


On 13.04.2020 01:18, Zebediah Figura wrote:
> So, I figure it's probably worth quantifying what we'd gain this way 
> (as well as how many libraries we'd potentially have to build or 
> maintain). As I see it, the libraries that can be built as PE (unless 
> I misunderstand what any one of them requires, so please correct me if 
> I'm wrong) are:


Some of those are not obvious and will need a separated discussion, but 
let's not distract from the main topic for now. I think that the list 
will look somewhat like this.


> On 2/14/20 12:43 PM, Esme Povirk (they/them) wrote:
>> Options for distributing dependencies:
>>   * Distribute headers and libs in a tarball built from wine-pedeps.
>> The binaries would be part of an addon, also built from wine-pedeps.
>
> I personally don't like this. I think there's value in
> (1) not requiring the user to install libraries they don't need (like, 
> say, libkrb5),


I think that fully functional Wine is the right default. If you think 
about addons as a part of Wine ecosystem, it's not much different from 
requiring the user to install jscript.dll even if they don't need it.


> (2) not prompting for every add-on (personally I feel that two is 
> already too many),


Sure, it would be nice to improve user experience with Wine addons. The 
reason I'm not motivated to work on it myself is that it's really just a 
fallback. If distro does its job well (and I think most of distros do), 
users don't need that anyway. IMHO we should concentrate on making it 
easy for distros to ship addons themselves, but fallback improvements 
are welcomed as well.


> (3) letting the distribution choose what to distribute by default.


Sure, it should be possible to disable things at build time. Just like 
it's already possible to disable jscript.dll at build time.


>
>>   * Rely on the host distribution to package mingw headers, libs, and
>> binaries for the dependencies we need. The binaries would then be
>> distributed with Wine. Fedora is the only distribution I'm aware of
>> that has an extensive library of mingw packages. To make this easier
>> on other distros, I could maintain wine-pedeps as an alternative
>> source of headers, libraries, and binaries at build time. The trouble
>> with this approach is that it's not clear how we can keep Wine's
>> libraries independent from the application.
>
> This was the idea that I proposed when the discussion came up 
> internally. It strikes me as the architecturally "correct" thing to 
> do. We're essentially building some of our files with a different host 
> architecture, so just as we'd link separately to x86_64-pc-linux-gnu 
> libraries and i686-pc-linux-gnu libraries, so too would we link 
> separately to x86_64-w64-mingw32 libraries (as well as i686-w64-mingw32).


Yes, some distros provide such packages and it's nice. I happened to 
contribute to that on mingw-w64 side. The purpose of those packages is, 
however, different than what we need for Wine. It's meant to provide a 
nice way for cross compiling projects with open source dependences. And 
once you cross compile such an application and want to ship it, you need 
to copy required DLLs or statically link dependencies. It's a very 
different goal than assembling Windows distro that we need.


What you propose is essentially forcing packagers to assembly and 
Windows (Wine) distro and I don't like that. Linux distros should 
concentrate on providing good native userspace and it's Wine's 
responsibility to use that to provide Windows-compatible environment. 
The fact that we will leverage PE build of some project should be 
something that packager can control, but he should not be forced to take 
care and maintain 20 packages only for Wine.


We definitely don't want such Wine dependences to be used in place of 
application provided DLLs with the same name. Renaming the library in 
build time seems like way cleaner way than any runtime heuristics that 
this thread led to.


Another thing where your solution can become a nightmare is ABI 
incompatibility. We don't control the dependencies and some of them are 
not ABI stable. We just need to accept it and make sure our solution 
takes that into account. So for example, when something like this 
happens (I don't think this is an isolated example, but I happen to know 
about it and it's hilarious: they had an incompatible MSVC and mingw 
ABI, so decided to break and uglify mingw ABI to follow MSVC and hack 
around a problem that I believe is imaginary):

https://bugzilla.libsdl.org/show_bug.cgi?id=4771

What I propose is to have a solution where we just bump just one 
constant in one repo, which would make Wine use wine-sdl2-2.dll instead 
of wine-sdl2-1.dll and that's it. All distros are be happy.

What you propose is that now tens of distros would need to notice the 
problem and express it in their package dependences. It means tens of 
action tickets spread around the ecosystem and a few Wine NOTOURBUG bugs 
reported upstream. And all of that work is just to provide a library 
that's otherwise internal to Wine anyway.


> I don't know what we'd do about Mac, but along the same lines, I'd 
> think we'd want to extend/analogize the current solution regarding 
> bitness to the whole architecture triple. 


Well Mac would be even easier to target than it is now if we provide 
right tools (because some of those dependences are tricky on Mac 
already). And if some packagers really insist on splitting those 
packages, they can still do that. with what I proposed. They can use our 
build system as a 'driver' or do it manually, that's all fine. We'd be 
not forcing one way or another.


So let me give a few examples of different choices packagers can do. I 
will call wine-core Wine as we know today (which will not change 
architecturally) and wine-meta-distro the proposed separated repo for 
dependencies build system allowing a simple and straightforward way to 
bootstrap complete Wine+dependences package, depending only on cross 
compilation tools.


1. Not care, provide just wine-core package (addon headers are just a 
build time dependency). Wine will download and cache addons in runtime.

2. Use wine-core the way distro currently do to provide Wine package. 
Use wine-meta-distro to build Wine addons as one but separated package.

3. Use wine-meta-distro to produce both a fresh wine-core and wine 
addons package in a single step for a single package.

4. Like 2 or 3, but distro wants some customization itself. They can do 
that, for example, by pointing wine-meta-distro to its own version of 
relevant package, pass additional config options or do simple source 
code modifications.

5. Split Wine into 20 packages. A distro can use wine-meta-distro to do 
build config stuff, so that they don't need to change their scripts 
every time something changes in any of dependent packages, but otherwise 
treat every dependency as a separated package.

6. Split into 20 packages and deal with them without wine-meta-distro 
packages. This will require packagers to redo all the work that 
packagers choosing 1-5 will be getting for free.


Your proposal seems to be similar to something between 5 and 6. I don't 
expect packagers to be thrilled with that idea if they have 2-4 as 
alternatives, by the key point is: let's give them a choice.



As a side note, a different and separated distro packages for, say, 
libxml2 built with winegcc could also have some use cases. That would be 
something potentially useful for cross compiling an application using 
winegcc, but not something for Wine to use internally.


Cheers,

Jacek