flexible-mmap breaks application

[Uwe Bonnes]

Hallo,

from the XILINX Ise suite (version 6.3 tested), the GUI (_pn.exe) and the
web update (webupdate.exe) crash with recent CVS when run on a 2.6 Kernel
(Suse 9.2, 2.6.11.1-20050307183958-default, but also the default 2.6
distribution kernel and a Athlon/ Pentium M). Both programs run fine with
2.4 (meeding a step back to Suse 9.0, but the same wine executable, only
tested on the Athlon).

The problem was hard to pinpoint, but I suspected differences in HeapAlloc()
return values to cause the problem, hinted by looking at the disassembly
where the error happens.

Clearly there is a difference between 2.4 and 2.6 kernels: 
While mmap2() in kernel 2.4 in map_view() returns values in the 0x4000 0000
range, kernel 2.6 returns values like 0xb7fe9000, 0x58710000, 0xa07f0000 and
0x77abf000. A check in RtlAllocateHeap() for return values above 0x80000000
also triggers with both programs when run on 2.6 and not with 2.4.

Looking deeper, I found http://lwn.net/Articles/92281/ about
flexible-mmap-update.patch and the wine-devel discussion from around new
year about "Re: ptrace single-stepping change breaks Wine".

Setting /proc/sys/vm/legacy_va_layout, like proposed by Ingo, lets the app
finally run. 

Alexandre finally concluded the discussion by
> Probably the easiest would be to have a way for an app to specify the
> mmap range it wants. So instead of having the kernel try to guess from
> brk and stack ulimit, both of which are meaningless for Win32 apps, we
> could set the range from "end of win32 exe" to 0x7ff0000. This would
> also avoid the need to preallocate everything above 0x80000000 that we
> currently do and that plays havoc with address space limits.

However nothing seems to have happened with regard to that problem until
now. Could we revive that discussion? 

Bye
-- 
Uwe Bonnes                bon at elektron.ikp.physik.tu-darmstadt.de

Institut fuer Kernphysik  Schlossgartenstrasse 9  64289 Darmstadt
--------- Tel. 06151 162516 -------- Fax. 06151 164321 ----------