address space layout

[Tijl Coosemans]

Hi all!

I had a user running FreeBSD 6.2, Xorg 7.3, ATI r200 DRI driver report
a problem where running Warcraft3 crashed because it ran out of malloc
heap space. The error was:

Assertion failed: (texObj->DriverData != NULL), function 
r200BindTexture, file r200_tex.c, line 1098.
fixme:ntdll:FILE_GetNtStatus Converting errno 12 to STATUS_UNSUCCESSFUL

This error happens inside the r200 driver, errno 12 means ENOMEM. There
was a bug report with a patch over at
http://bugs.freedesktop.org/show_bug.cgi?id=12184

The patch gets rid of the assertion failure by doing proper error
checking, but it doesn't solve the underlying problem of course,
causing a NULL pointer dereference in Wine somewhat later:

wine: Unhandled page fault on read access to 0x00000058 at address 
0x7e99cc84 (thread 0009), starting debugger...

This isn't the first time such a problem appears, especially with games,
so ultimately a solution has to be found.

The problem is that the way Wine wants to setup the address space is
really somewhat incompatible with the way FreeBSD mmap and malloc work.
FreeBSD mmap allocates from the end of a process data segment upwards
towards the stack. Linux mmap allocates from the stack downwards.
FreeBSD (<7.x) malloc doesn't fall back to mmap when it runs out of
space in the data segment. Linux malloc does (I've been told).

Because the wine process loads itself at 0x7bf00000, this means on
FreeBSD with mmap going up, both the data segment and all shared
objects (like builtin dlls) sit squeezed between that address and
0x80000000 since addresses above 0x80000000 can't be used for builtin
dlls. It's about 64Mb which is currently split 50/50 between heap space
and shared objects. This works in a lot of cases, but sometimes a 32Mb
heap isn't enough like in the example above, where I assume it's been
taken up by textures and other graphics data.

Linux doesn't have this problem because mmap allocates shared objects
from 0x80000000 downwards and can go beyond the address where wine is
loaded if necessary. It can allocate extra heap space this way too when
needed.

Alexandre and I discussed this a couple weeks ago, but didn't really
come up with anything. The problem back then (WoW) has since gone away
(turned out to be a memory leak somewhere) so everything was left the
way it was. Now it has turned up again however with a different program
on a different setup, so it's something that really needs to be looked
at. It could again be a memory leak in some other project's code, but
then that just means the current situation isn't robust enough to deal
with that, since it all does work on Linux.

It appears that the only solution is to locate wine somewhere after
0x80000000 instead of at 0x7bf00000 and to allow mmap to allocate
memory there. Then the data segment size limit can be set to 256Mb or
something instead of 32Mb. The problem is then that builtin dlls are
mmapped (by the runtime linker) well beyond 0x80000000. Now I'm
thinking, would it be possible to load dummy dlls somewhere in the
first 2Gb of address space? Through which API calls can a program
determine a dll is above 0x80000000 by the way?