World Wine News

Happy new year everybody!

Being the patch fetishist that I am, I couldn't help noticing that in December 2004 we cleared 8MB of patches: that's by far the largest since the current archives began in October 2000.

The extra traffic came from a lot of places:

• Huge improvements to the Application Database.
• The number was also pushed higher by an MSI merge from CodeWeavers.
• Some big janitorial patches went in from Michael Stefaniuc
• Finally Rob and I were cranking out DCOM patches, and there are of course lots more in the pipeline.

Now time for some crystal ball gazing. Here are some interesting patches we might be seeing in 2005:

• Completion of the WM rewrite work. Alexandre seems to be making great progress on this, which will let us fix many of the odd visual corruption problems we've been plagued with for so long. It'll also let us remerge Alex's winedesktop patch, giving us a real desktop window that is used for all apps, at last.

  It should have many other benefits too: fixing the flickering in the Half Life menus and allowing us to support the NETWM fullscreen protocol. This should let us resolve many instances of the "my game starts but I can't type" problem.

• Lots more DCOM code. Huw Davies is hacking on getting widl to produce a stdole32.tlb file, once that's in we can start really nailing InstallShield to the wall. We will hopefully get this and the thread-affinity patches in the next few months, which should mean for the first time we can run InstallShield 6 installers perfectly, out of the box.

  Support for newer InstallShields will come as time permits of course.

• More MSI improvements! CodeWeavers are continuing to work on this so it should develop nicely even though the iTunes installer work has been completed. Office 2003 will also motivate this. It's being used by at least one commercial app porting project as well.
• Support for running Winelib apps directly, so we can get rid of the .exe.so extensions/shell wrapper scripts, and produce "raw" ELF binaries. They'll still need the wine loader app to be present of course, but it should all be invisible and behind the scenes. Vincent Beron and I have been working on this in the last week or so, and we just need to finish it off and convince Alexandre the extra assembly is worth it! :)

Who knows what the new year will bring? More apps working out of the box, that's for sure! Have fun!

For those that like such statistics:

http://home.wanadoo.nl/wijn/wine/cvssize.png

I thought I'd post an update on where Rob and I are up to, and where we're going. This is the plan that I have in my head, Rob may want to do things in a slightly different order etc so don't be surprised if we change some stuff around as a result of this email.

We're currently reworking, extending and in some cases rewriting the code Marcus wrote originally to work more like native DCOM does. This means that new features and major improvements are *not* going to happen for a while, because before we can extend our support for DCOM we have to migrate to the same architecture that Microsoft uses internally. This has several benefits:

1. It saves us design work, because MS already did it
2. It's the only way to be wire compatible in the long run
3. It means the documentation (like Essential COM etc) is already written

Getting to a more native-style architecture has already been started, but won't be finished for a while. Here are the steps we need to take, the ones marked with a * are already done, - [ed note: or a bullet ] means they are still to do.

• Replace the wine_marshal_id structure with [STD]OBJREF. This is necessary because the Microsoft internal interfaces like IOXIDResolver and IRemoteActivation operate in terms of OBJREFs. An OBJREF is a marshalled interface pointer, so by using the same structure we're able to be wire-compatible with MS.
  * Switch apartments to be identified by OXIDs
  * Refactor and extend the stub manager, identified by OIDs
  * Refactor and extend the proxy manager
  - Make all system interface proxies support aggregation
  - Identify interface stubs by IPID not IID
  - Do a search and replace of wine_marshal_id with STDOBJREF
• Alexandre has stated that he won't allow any more midl generated code in WineHQ. Enhance widl until it can generate raw RPC marshallers for at least IRemUnknown.
• Generate unique names for the pipes we currently use in communication and pull the right name from the OBJREF DUALSTRINGARRAY. In real DCOM this contains the RPC string binding for the OXID Resolver endpoint. This should be the first enhanced feature, as at this point you can run two apps at once that happen to marshal the same interface, and they won't interfere.
• Implement IRemUnknown, using custom marshalling for now. Use this to nail distributed refcounting, at the moment we have a bunch of bad hacks for that.
• Flip DCOM on top of the RPC transport layer, replacing the current named pipe based IRpcChannelBuffer implementation. Not necessarily using NDR marshalling at this point, though we may find it's necessary (?)
• Rework RPC dispatch to be more Windows-like, so call re-entrancy works.
• Implement RPC thread affinity. This will fix the InstallShield painting problems ... yes it's a long way off :(
• Begin rewriting/moving the typelib marshaller to rpcrt4 and to use NDR (in Windows this is done by generating MOPs at runtime from the type data)
• .... this point is so far off I don't really care about it ....

I probably forgot some items as well. But as you can see we only just begun. Also remember that Rob may want to modify it and also we may change direction at any point to make stuff we want internally work (like native MSIv2 ...)

This adds preliminary support for typelib generation to widl. It's not enough to generate stdole32.tlb mainly because it doesn't cope with some of the builtin types, like VARIANT, properly yet. However it should be possible to generate simple typelibs (for testing the typelib marshaller for instance). I'm going to continue to work on this so let me know if you're interested in helping out.

Somebody removed dcom95.exe from sf.net. Was there a reason for that? We really need to keep that file mirrored. It's an essential file for many programs to run under wine.

I've re-added the file as a hidden file for the moment. It won't show up in the download list, but we can still link to it.

Really Ivan, you may be technically in the right here, but the fact is you should absolutely have discussed this on wine-devel first. That file is referenced by various third party scripts/howtos and such, I know this because I've written these things before. By taking this matter into your own hands and not discussing it first you went behind the backs of the community and broke a lot of code and documentation by doing so. My IE installer script is one example iirc, but I'm sure there are others.

Your admin access on Sourceforge is so you can upload the Mandrake RPMs, it is *not* so you can interfere with decisions that were made a long time ago at will. Please remember that in future.

I was puzzled by this, because I shared Mike's concern that you unilaterally removed this file, and it seemed abrupt and unannounced to me as well.

So I searched around, and I see that you brought this up in April, and then nicely followed through by submitting patches to change Wine to not point to SF, but instead to Microsoft.

So, I guess that the only thing we could have asked is if you had posted a note saying something to the effect 'okay, guys, now that I cleaned up the code to not point to SF, I'm removing the file now...', which would have wacked us with the clue bat. And it's a bit tough to ask us to remember things 8 months out of context...

But it is clear to me that you did try to do 'the right thing', and we jumped down your throat a bit too quickly.

Sorry about that.

I'm afraid Alexandre has decided not to apply this patch (the ABI personality syscall). His reasoning is as follows:

• There are a million kernel patches out there which use the ABI personality system to break stuff, which means that realistically we have no idea what it does
• Therefore, using it would be a big support problem as it may even switch off stuff (new features etc) we want, so this system doesn't really scale
• It'd be a lot better to provide a "disable this mmap feature/behaviour" flag for each thing that changes it, or *even* better to make it opt-in as having to change the sources to make old apps work again kind of defeats the point of backwards compatibility (ie, old apps should continue to work as-is)
• Another possibility would be to create a new mmap API that lets us ask for exactly what we want, instead of second-guessing the kernel. I don't know exactly what sort of an API Alexandre has in mind here, perhaps he could describe it.
• So rather than apply the patch, we have to figure out why flex-mmap is breaking this program and add yet more magic VM code to stop it from happening :(

Could you upload a +relay,+tid,+seh,+msgbox trace somewhere please? Of course if you could investigate it yourself that'd be the best thing.

I wasn't sure whether to CC some kernel people or not, but it seems from previous threads that the ABI personality syscall system is meant for people like us who are trying to keep "legacy" binaries working. Unfortunately this sort of unbreak-me-please flag isn't really what we need/want from the kernel.

Also a precise description of what flex-mmap does would be good. Google wasn't too informative, best I could get was that it means mmap allocates from the "top" of the address space down. But where is the top exactly?

Ingo has described it thus:

before:

0x08000000 ... binary code
0x08xxxxxx ... brk area
0x40000000 ... start of mmap, new mmaps go after old ones
0xbfxxxxxx ... stack

after:

0x08000000 ... binary code
0x08xxxxxx ... brk area
0xbfxxxxxx ... _end_ of all mmaps, new mmaps go below old ones
0xbfyyyyyy ... stack

the 'after' layout guarantees that brk area (malloc()) can grow unlimited and mmap() can grow unlimited - they will meet somewhere inbetween when almost all of the VM is used up. [the 'top' of the mmaps in the 'after' layout is constrained by the stack ulimit - the stack must still fit and we never allocate into the stack's yet unallocated and growable hole.]

with the 'before' layout we've got 900 MB for brk() and 1.9GB for mmaps() - a rigid limit.

another workaround to switch off flex-mmap is to set the stack ulimit to 'unlimited':

saturn:~> cat /proc/self/maps | tail -7
b7db3000-b7db4000 r--p 00cfd000 03:41 3735973 /usr/lib/locale/locale-archive
b7db4000-b7de1000 r--p 00ccc000 03:41 3735973 /usr/lib/locale/locale-archive
b7de1000-b7de7000 r--p 00cc3000 03:41 3735973 /usr/lib/locale/locale-archive
b7de7000-b7fe7000 r--p 00000000 03:41 3735973 /usr/lib/locale/locale-archive
b7fe7000-b7fe8000 rw-p b7fe7000 00:00 0
bff2c000-c0000000 rw-p bff2c000 00:00 0
ffffe000-fffff000 ---p 00000000 00:00 0

saturn:~> ulimit -s unlimited

saturn:~> cat /proc/self/maps | tail -7
42188000-4218a000 rw-p 00014000 03:41 3433982 /lib/ld-2.3.3.so
4218c000-422aa000 r-xp 00000000 03:41 3434006 /lib/tls/libc-2.3.3.so
422aa000-422ac000 r--p 0011d000 03:41 3434006 /lib/tls/libc-2.3.3.so
422ac000-422ae000 rw-p 0011f000 03:41 3434006 /lib/tls/libc-2.3.3.so
422ae000-422b0000 rw-p 422ae000 00:00 0
bfea0000-c0000000 rw-p bfea0000 00:00 0
ffffe000-fffff000 ---p 00000000 00:00 0

e.g. SuSE defaults to an unlimited stack so flex-mmap is effectively disabled there.

To set the VM to legacy, for all apps, set /proc/sys/vm/legacy_va_layout to 1.

Probably the easiest would be to have a way for an app to specify the mmap range it wants. So instead of having the kernel try to guess from brk and stack ulimit, both of which are meaningless for Win32 apps, we could set the range from "end of win32 exe" to 0x7ff0000. This would also avoid the need to preallocate everything above 0x80000000 that we currently do and that plays havoc with address space limits.