linux mmap bug ? not filling in allocation gaps

[Geoff Brimhall]

I've been debugging why an n64 emulator ( Project64 )
gets "Failed to allocate memory" errors under wine
9.9.10 on linux kernel 2.6. 

The program allocates a large chunk of size
0x20100000, which succeeds, and then another large
0x10100000 chunk, which fails.

WINEDEBUG=+relay,+virtual eventually brought me to
look at the linux /proc/<pid>/maps to see if the
address space was really fully allocated. Here's what
I noticed:

Address start - end:
00400000-0554000 - Project64.exe address
00554000-20000000 ---p large mem allocation
*** <gap> ***
55555000-573e0000 - loaded shared libraries
573e0000-774e0000 ---p mem alloc of 0x20100000
7bf00000-7c003000 - wine-pthread
7c003000-7c1ee000 - [heap]
7ffe0000-fffe0000  ---p large mem allocation
ffff7000-fffff000 - [stack]

Wrote a test mmap binary to fill a process address
space, it created a similar gap between the executable
load address and the loaded shared libraries,
eventually failing mmap calls when it hit the stack
address space. When I updated the test app to
explicitly mmap in the gap area via specifying the
aligned start address, the mmap call succeeded.

I can't be the first person to notice this. Are there
any TODO solutions for this ?

One linux-centric solution would be to look for gaps
like this in /proc/<pid>/maps in the libs/wine/mmap.c
and take advantage of this. Or maybe another would be
to keep a table of address spaces returned in
wine_anon_mmap as a guide, then validate with
try_mmap_fixed as to where a next-allocation should
occur ( would this solution work outside linux ) ?

Thanks,
geoff Brimhall

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com