X86/PPC linking (was Re: [Darwine] Re: Wine Emulation: Swapping functions)

[Jim White]

N.B. The only substantive paragraph here is the last one, so folks 
interested in the X86/PPC linking problem should just jump down there.

Mike Hearn wrote:

>> I agree that flipping the syscalls is a much more promising approach 
>> than the WIN32 API.  I'm also thinking that there is bound to be some 
>> relevant code around aleady (perhaps even in QEMU).
> 
> Yes QEMU can do syscall conversion iirc, at least in some modes.
> 
>> While the current phase of the Darwine effort is concerned with making 
>> something that works, naturally performance is an important issue.  In 
>> fact one of the architectural motivations for Darwine is that it will 
>> be able to achieve much higher performance than approaches that 
>> virtualize the OS (in addition to the objectives of not having to run 
>> Windows).
> 
> I don't think you could get much higher performance than eg, Virtual PC 
> to be honest. Maybe if you developed some way of pre-converting the 
> binaries beforehand rather than doing it on the fly, but VPC type 
> systems have the advantage of having very limited interaction points 
> between the host OS and the client OS which can be tightly optimized 
> (eg, video ram), so they can perform a lot of tricks you can't do if you 
> want to integrate better with the OS.

Obviously I completely disagree.  Such tweaking at the device layer is 
stuck with emulating all the code all the time plus driver stacking and 
will never achieve native performance.  Darwine's plan for integrated 
emulation means that it has the potential for native performance with 
the only penalty being for application code which must be emulated.  And 
because we change the loader it can be precompiled delivering 
performance which could be several times faster than what a VPC/QEMU OS 
(or user mode) virtualization can do.

>>  > I think they were only converting the syscalls but I don't really
>>  > remember .... Gav told me various stories of their Mac porting efforts
>>  > at WineConf.
>>
>> It would be interesting to chat with those folks indeed.  I take it 
>> that given that their product is dependent on Wine, which is GPL, the 
>> relevant code which enables it to run on Mac OS X must be available as 
>> well?  That would dramatically simplify what we need to do!
> 
> Please do some basic research, otherwise you just come off sounding 
> ignorant:

Of course I'm ignorant of the most everything regarding Wine history. 
How is that germane to anything?

Your hostile attitude is why I try to limit Darwine discussion on 
wine-devel only to wine-specific issues.

> a) Wine is not under the GPL and never was, it's under the LGPL

Oh please.

> b) TransGamings Cedega forked from Wine several years ago, back when it 
> was under the X11 license which allowed that. So they don't have to 
> contribute their code back.
> c) That fork was what motivated the switch to LGPL.

Well that makes sense.  I couldn't make heads or tails of their 
licensing.  Although it does say something about the Wine of WineX 2 
being LGPL.  In any case it seemed clear that whatever they had is not 
free.  Also I can't see why they would want to do any X86 emulation at 
all since games do usually need maximum performance.  I have to figure 
that they are using Winelib.

Of course as you say I am ignorant of the facts, although I would be 
delighted if any Transgaming or related folks would help with Darwine.

> d) Quite a few of TransGamings porting projects don't seem to involve 
> Wine at any level. For instance Gav told me about a game they ported 
> which used Java.
> 
> Intererestingly, I think Gav thought the Java port was *harder* than 
> Wine ports were due to the huge instability of Java on the Mac platform. 
> He said at one point Apple pushed out an online update to Java which 
> broke their game and then withdrew it, leaving no way for them to 
> duplicate the environment that their customers were having problems with 
> (amongst other problems)! So much for "write once, run anywhere" ...

That was indeed true before OS X.  Typically ironic/tragic of Apple that 
the machines most in need of WORA had lousy Java.  Fortunately on OS X 
the Java is at least as good and in some cases (such as Swing 
performance) is better than on Windows.

>> Hmmm, not sure what you mean there.  Initially we patched winegcc to 
>> make the OS X API accessible, but at Alexandre's suggestion to Pierre 
>> we've switched to binding to the symbols dynamically.  Of course using 
>> the native Carbon API is necessary, otherwise we're stuck in the X11 
>> box which is as foreign to the Mac user as having Windows running via 
>> VPC.
> 
> You realise that a quartzdrv would not change the fact that Windows apps 
> look like Windows apps, right?

This is another area that integration opens up for innovation.

> All it changes is the way the bits drawn by Wine get to the screen. To 
> be honest I think you'd achieve much better results by improving the 
> quartzwm so X11 windows integrate better with the dock etc. If quartzwm 
> is proprietary (don't recall) then writing a new WM that is equivalent 
> and has all the features you need would be another way forward.
> 
> While it's interesting from a technical perspective to do a quartzdrv, 
> I'm really not convinced you would get better integration than just 
> improving the X11 support in the OS, which would obviously also benefit 
> those who use Linux/X11 apps on the Mac as well as Wine users.

I agree that a really good WM for OS X X11 could make X11 applications 
almost behave like decent Mac applications.  I'm sure that there are 
folks who like X11 and OS X working on it.  Apple's support for X11 is a 
delightful surprise.

Of course even so, as I said to begin with, such driver layering is 
devastating to performance and any hope of order-of-magnitude speed up.

> Anyway .. what I meant by that statement is that if you're running x86 
> Wine in QEMU you can't access the Carbon APIs directly. I think in the 
> mode where you run foreign binaries QEMU will perform syscall conversion 
> but obviously this is of limited use - the OS X APIs aren't accessible 
> via syscalls.

Yes, quite so.

> One thing you could do I guess is expose each OS X API that you wanted 
> as a new "fake" syscall and then use a customized version of QEMU that 
> converted it into a Mac API call, ie marshal the calls in and out of the 
> emulated environment. You'd still pay a performance penalty but that's 
> unavoidable. I'm not sure how you'd deal with callbacks.

The ideal method would be to handle this with a trick loader that can 
link to PPC code which is escaped from the X86 emulation.  We can 
automatically generate X86 DLLs for the PPC dynamic libraries (and hence 
the OS X APIs) which have little emulation escape thunks in them.  There 
is zero performance penalty since the emulator can compile such escapes 
as direct native calls.  Callbacks use a similar sort of little thunk 
which calls the emulator with a given X86 address.

I would really like to get away from having to write all the little 
wrappers to deal with the namespace problem.  I believe we can solve 
that with a suitable naming convention as we began to do with the 
winegcc patch.

Alternatively we could go with with dynamic binding as proposed by 
Alexandre and have a syscall (or emulation escape) to call PPC code. 
This is probably the easiest to implement but leaves us with extra call 
overhead (although apparently this is something that is currently done 
in Wine and is quite small).

By using suitable macros we can make the source independent of which 
method is used.

The thing we don't get away from is that any code which calls OS X 
routines from X86 code *will* have to deal with endianess.  There is 
overhead here, but we can be careful to use code that the emulator does 
a good job translating.  Of course if there is substantial code (which 
would be unusual) it can simply be compiled as a PPC dynamic library.

As for swapping functions, the preceding discussion reiterates my belief 
that we should confine such fussy business to the relatively small bit 
of code that calls into OS X, will be relatively stable over the long 
haul, and over which we have full control.  And as for the concern over 
having Wine be X86 and hence incurring emulation overhead, this would be 
one of the first bits of code which would be a candidate for having its 
emulator-compiled code cached.

Jim White