Request for discussion: Using PE libraries for Wine dependencies

[Jacek Caban]

On 15.04.2020 21:29, Zebediah Figura wrote:
> So I think I may have misunderstood or lost sight of goals somewhere 
> along the way, let me try to review.
>
> I think it makes the most sense to use one or more separate 
> distribution packages for our PE dependencies, rather than 
> distributing them as part of Wine, statically linking, or distributing 
> them using our Win32-based addons system. I'm not sure just from 
> reading this thread there's a general consensus regarding that, but 
> I'm not sure I've seen anyone argue against it either.
>
> Separately, I think it makes sense for Wine's libraries—seeing as they 
> *are* (relatively, and mostly) high-level—to use MinGW libraries the 
> same as any other MinGW-compiled program would.
>
> It's not clear to me that this implies that we reverse the direction 
> we're currently taking and try to make more use of the MinGW CRT, 
> headers, and/or libraries. (I can understand the CRT part, though even 
> then I have to wonder if there are differences that will matter.)
>
> If it does, I'd like to understand why that's not possible, because it 
> seems to me that we gave up on MinGW after they objected to taking the 
> entire mingw-w64 project under the umbrella of Wine


Let's make it clear: there is no give up. Cooperation goes on, just like 
it did before. It just didn't change. We continue to use mingw to build 
Wine, we just don't use a few static libraries that we didn't need anyway.


To give you an example for the context, look at another prominent 
mingw-w64: Cygwin. For obvious reason they are interested in Win32 
platform and use mingw-w64 for platform headers and importlibs. But they 
have their own ABI, for which they provide their own C runtime 
libraries. mingw-w64 allows that, the build system has special features 
to make such things possible. Cygwin also offers mingw-w64 in a form of 
a cross compiler, which cross compiles from Cygwin ABI as a host to 
mingw ABI target (obviously using mingw crt). It's an extreme example 
why "PE is all the same" in not true.


That analogy also applies to Wine in a more delicate way. 
msvcrt/ucrtbase implements a crt for msvc ABI, which is very similar, 
but different than mingw ABI. Wine needs to provide msvcrt anyway for 
obvious reasons and that's about all we need as for crt.


mingw uses ABI that is slightly different, which has quite a few 
implications for crt. It's a mixture of static libs and msvcrt importlib 
with some features that are less than obvious. For example mingw has a 
real long double which on msvc is the same as double. For that reason, 
mingw-w64 crt can't use msvcrt long double functions, but instead 
provides its own implementations of relevant functions in a static lib. 
If you want full support for long double, you may even choose to use 
*printf functions provided by mingw-w64 that support long double formats 
(and also other standard compliance, which was handy at a time, but not 
so much with ucrt around).


long double is obviously not a blocker, it's just an example to give you 
high level understanding what mingw-w64 crt really is and why. Now, look 
what fully using mingw-w64 for crt means part by part:

- mingw-w64 crt headers.

Wine has its own msvcrt headers that could not be blindly just replaced. 
For example we need those headers to work on non-mingw compilers (like 
for ELF builds). We'd need make mingw-w64 worry about platform it 
otherwise doesn't need to care. It's doable, but it's not clear if worth it.

- mingw-w64 msvcrt importlib

As explained at importlib level a lot of functions are implemented in 
static lib instead of importing from DLL. We provide importlib in form 
of .spec file anyway.

- standard compatibility stuff

mingw-w64 has a number of features providing standard compliance in 
places where msvcrt did bad job. It was handy in the past for msvcrt 
version that were rather bad. It's not so useful on top of ucrt.

- compatibility stuff

A lot of functions are in static lib for compatibility with old msvcrt 
versions that didn't provide them. A lot of those are just manual delay 
loads with some static fallbacks. We don't need those.

- more advanced startup files, a few features that really belong to 
static lib like atexit

We don't need any of those for Wine itself. We may need some in the future.


I hope that answers your question about crt part. If you still think 
there is a problem, please be more concrete.


That said, there is definitely room for improvements. I happen to be the 
one maintaining sharing that actually happens for years (which is why I 
find it ironic that you complain to me about lack of sharing) and I plan 
to continue doing so. Let me use a chance to raise awareness in case 
someone wants to help. mingw-w64 semi-automatically imports a number of 
headers and widl from Wine tree. On mingw-w64 side there is assumption 
that for imported stuff, Wine review replaces mingw-w64 review process. 
mingw-w64 has two import scripts that are really basic and trivial to 
maintain scripts doing the job. If you'd prefer to do it differently, 
patches welcomed.


We could share a lot more platform headers, but it's a real work. In 
case a header is missing in mingw-w64, it's a matter of testing and 
making sure all its dependences are fine. If it's already duplicated 
between projects, then deduplication needs a review to make sure that 
one of copies is complete enough to replace the other by sending a patch 
to Wine. Once it's in Wine, send a patch to import script to mingw-w64. 
 From this moment, both repos will stay synced. If you hypothetically 
want fully merged platform headers in the future, this is the way to do 
it incrementally. And it's already there.


widl could also use some work. The most notable example that sees demand 
on mingw-w64 side is better winrt support.


If you really want to help mingw-w64 and Wine, I think that contributing 
to any of above is a good way to do it.


> Now, on the assumption that using mingw-w64-compiled PE libraries is a 
> reasonable thing to do...
>
> On 4/13/20 11:38 AM, Jacek Caban wrote:
>> So let me give a few examples of different choices packagers can do. 
>> I will call wine-core Wine as we know today (which will not change 
>> architecturally) and wine-meta-distro the proposed separated repo for 
>> dependencies build system allowing a simple and straightforward way 
>> to bootstrap complete Wine+dependences package, depending only on 
>> cross compilation tools.
>>
>>
>> 1. Not care, provide just wine-core package (addon headers are just a 
>> build time dependency). Wine will download and cache addons in runtime.
>>
>> 2. Use wine-core the way distro currently do to provide Wine package. 
>> Use wine-meta-distro to build Wine addons as one but separated package.
>>
>> 3. Use wine-meta-distro to produce both a fresh wine-core and wine 
>> addons package in a single step for a single package.
>>
>> 4. Like 2 or 3, but distro wants some customization itself. They can 
>> do that, for example, by pointing wine-meta-distro to its own version 
>> of relevant package, pass additional config options or do simple 
>> source code modifications.
>>
>> 5. Split Wine into 20 packages. A distro can use wine-meta-distro to 
>> do build config stuff, so that they don't need to change their 
>> scripts every time something changes in any of dependent packages, 
>> but otherwise treat every dependency as a separated package.
>>
>> 6. Split into 20 packages and deal with them without wine-meta-distro 
>> packages. This will require packagers to redo all the work that 
>> packagers choosing 1-5 will be getting for free.
>> Your proposal seems to be similar to something between 5 and 6. I 
>> don't expect packagers to be thrilled with that idea if they have 2-4 
>> as alternatives, by the key point is: let's give them a choice.
>>
>>
>>
>> As a side note, a different and separated distro packages for, say, 
>> libxml2 built with winegcc could also have some use cases. That would 
>> be something potentially useful for cross compiling an application 
>> using winegcc, but not something for Wine to use internally.
>
> I think 2/4 are referring to the same thing (at any rate, I see no 
> reason not to give distributions the choice). I don't think I fully 
> understand how 3 differs from 2/4, or how 6 differs from 5. Do you 
> mean by introducing a meta-package which only exists for dependency 
> purposes?
>
> Anyway, if depending on mingw-w64 really is a dead end, I guess either 
> 4 or 5 seems reasonable to me, though it's less clear that 
> distributions actually would prefer 4. In particular, I notice:
>
> * Suppose you want the versions of the PE libraries you ship to match 
> the ELF libraries your distribution already ships. You can change the 
> sources without too much difficulty, but how do you label the 
> resulting package with regard to version? Naming it after the Wine 
> version isn't great, especially if you're going to update those 
> packages asynchronously...


It's not very important implementation detail that we may decide once we 
have it sorted out on higher level.


> * If a distribution (or even WineHQ) wants to update one of our 20+ 
> packages, do they/we have to ship the whole wine-meta-distro again? 
> (Maybe this isn't a problem, if the packages in question support 
> partial updates...)


Well, just like we'd update jscript.dll in Wine installation even if it 
didn't change since last update.


>
> Ultimately I think this is something that actual distributors would 
> have to answer, and I think we should probably bring some in on this 
> thread.
>
>
> If on the other hand we can depend on mingw-w64 libraries, I think we 
> almost have to pick 5 over 4, for the simple reason that it's already 
> done that way. There's an existing demand for mingw-w64 libraries 
> which are distributed separately. Fedora already ships them. Arch 
> doesn't (in fact, Arch doesn't even ship the mingw-w64 compiler), but 
> they exist in the AUR as separate packages, and I myself use a 
> third-party repository that ships mingw-w64 library blobs. (That's how 
> I was able to write that msxml3 patch, in fact.) Debian/Ubuntu don't 
> ship very many, but they do ship libz, and also development versions 
> of a few other libraries. 


Let me give a high level summary how I see it. I'm not attached to what 
I proposed, but here is how I can see the situation:


We're going to be shipping PE build ourselves one way or another (all 
solutions assume that so far). Whatever form that will take, we should 
make sure that it's something reusable for others. By ensuring that 
whatever we do is easy to reuse, we can make sure that forks and distros 
have the ability to do the same without duplicating efforts.


I think that the solution should be portable. Limiting it to a distro or 
a set of distros is not what I consider as portable. We will want to 
provide distro packages, just like we provide Wine packages. But I think 
that actual maintenance should be happening in platform-independent 
place. Just like Wine has no bits specific to any package manager in its 
sources.


It seems to me that autoconf and make provide tools to achieve that.


Jacek