<P>Hi Mike</P>
<P>I've started to revisit the architecture.sgml file, and started to write also about this matters (and a few others).</P>
<P>For coherency reasons, if you don't mind, I'll reintegrate your bits in the architecture.sgml file (what you wrote covers also process image loading...).<BR><BR>A+<BR><BR><BR></P>
<BLOCKQUOTE style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #ff0000 2px solid">> Message du 28/05/04 01:35<BR>> De : "Mike Hearn" <MH@CODEWEAVERS.COM><BR>> A : wine-patches@winehq.com<BR>> Copie à : <BR>> Objet : Add documentation on the address space layout in Wine<BR>> Mike Hearn <MH@CODEWEAVERS.COM><BR>> Add documentation on the address space layout in Wine<BR>> <BR>> Generated from:<BR>> * mike@navi.cx--2004/wine--mainline--0.9--patch-25<BR>> <BR>> --- /dev/null 2003-09-15 14:40:47.000000000 +0100<BR>> +++ documentation/address-space.sgml 2004-05-28 00:26:06.000000000 +0100<BR>> @@ -0,0 +1,175 @@<BR>> +<CHAPTER id=address-space><BR>> + <BR>> +<BR>> + <PARA><BR>> + Every Win32 process in Wine has its own dedicated native process on the host system, and<BR>> + therefore its own address space. This section explores the layout of the Windows address space<BR>> + and how it is emulated.<BR>> + </PARA><BR>> +<BR>> + <PARA><BR>> + Firstly, a quick recap of how virtual memory works. Physical memory in RAM chips is split<BR>> + into <EMPHASIS>frames</EMPHASIS>, and the memory that each process sees is split<BR>> + into <EMPHASIS>pages</EMPHASIS>. Each process has its own 4 gigabytes of address space (4gig<BR>> + being the maximum space addressable with a 32 bit pointer). Pages can be mapped or unmapped:<BR>> + attempts to access an unmapped page cause an EXCEPTION_ACCESS_VIOLATION which has the<BR>> + easily recognizable code of 0xC0000005. Any page can be mapped to any frame, therefore you can<BR>> + have multiple addresses which actually "contain" the same memory. Pages can also be mapped to<BR>> + things like files or swap space, in which case accessing that page will cause a disk access to<BR>> + read the contents into a free frame.<BR>> + </PARA><BR>> +<BR>> + <SECT1><BR>> + <BR>> + <BR>> + <PARA><BR>> + When a Win32 process starts, it does not have a clear address space to use as it pleases. Many pages<BR>> + are already mapped by the operating system. In particular, the EXE file itself and any DLLs it<BR>> + needs are mapped into memory, and space has been reserved for the stack and a couple of heaps<BR>> + (zones used to allocate memory to the app from). Some of these things need to be at a fixed<BR>> + address, and others can be placed anywhere.<BR>> + </PARA><BR>> +<BR>> + <PARA><BR>> + The EXE file itself is almost always mapped at address 0x400000 and up: indeed, most EXEs have<BR>> + their relocation records stripped which means they must be loaded at their base address and<BR>> + cannot be loaded at any other address.<BR>> + </PARA><BR>> +<BR>> + <PARA><BR>> + DLLs are internally much the same as EXE files but they have relocation records, which means<BR>> + that they can be mapped at any address in the address space. Remember we are not dealing with<BR>> + physical memory here, but rather virtual memory which is different for each<BR>> + process. Therefore OLEAUT32.DLL may be loaded at one address in one process, and a totally<BR>> + different one in another. Ensuring all the functions loaded into memory can find each other<BR>> + is the job of the Windows dynamic linker, which is a part of NTDLL.<BR>> + </PARA><BR>> +<BR>> + <PARA><BR>> + So, we have the EXE and its DLLs mapped into memory. Two other very important regions also<BR>> + exist: the stack and the process heap. The process heap is simply the equivalent of the libc<BR>> + malloc arena on UNIX: it's a region of memory managed by the OS which malloc/HeapAlloc<BR>> + partitions and hands out to the application. Windows applications can create several heaps but<BR>> + the process heap always exists. It's created as part of process initialization in<BR>> + dlls/ntdll/thread.c:thread_init().<BR>> + </PARA><BR>> +<BR>> + <PARA><BR>> + There is another heap created as part of process startup, the so-called shared or system<BR>> + heap. This is an undocumented service that exists only on Windows 9x: it is implemented in<BR>> + Wine so native win9x DLLs can be used. The shared heap is unusual in that anything allocated<BR>> + from it will be visible in every other process. This heap is always created at the<BR>> + SYSTEM_HEAP_BASE address or 0x65430000 and defaults to a megabyte in size.<BR>> + </PARA><BR>> +<BR>> + <PARA><BR>> + So far we've assumed the entire 4 gigs of address space is available for the application. In<BR>> + fact that's not so: only the lower 2 gigs are available, the upper 2 gigs are on Windows NT<BR>> + used by the operating system and hold the kernel (from 0x80000000). Why is the kernel mapped<BR>> + into every address space? Mostly for performance: while it's possible to give the kernel its<BR>> + own address space too - this is what Ingo Molnars 4G/4G VM split patch does for Linux - it<BR>> + requires that every system call into the kernel switches address space. As that is a fairly<BR>> + expensive operation (requires flushing the translation lookaside buffers etc) and syscalls are<BR>> + made frequently it's best avoided by keeping the kernel mapped at a constant position in every<BR>> + processes address space.<BR>> + </PARA><BR>> +<BR>> + <PARA><BR>> + On Windows 9x, in fact only the upper gigabyte (0xC0000000 and up) is used by the kernel, the<BR>> + region from 2 to 3 gigs is a shared area used for loading system DLLs and for file<BR>> + mappings. The bottom 2 gigs on both NT and 9x are available for the programs memory allocation<BR>> + and stack.<BR>> + </PARA><BR>> +<BR>> + <PARA><BR>> + There are a few other magic locations. The bottom 64k of memory is deliberately left unmapped<BR>> + to catch null pointer dereferences. The region from 64k to 4mb are reserved for DOS<BR>> + compatibility and contain various DOS data structures. Finally, the address space also<BR>> + contains mappings for the Wine binary itself, any native libaries Wine is using, the glibc<BR>> + malloc arena and so on.<BR>> + </PARA><BR>> + <BR>> + </SECT1><BR>> +<BR>> + <SECT1><BR>> + <BR>> +<BR>> + <PARA><BR>> + Up until about the start of 2004, the Linux address space very much resembled the Windows 9x<BR>> + layout: the kernel sat in the top gigabyte, the bottom pages were unmapped to catch null<BR>> + pointer dereferences, and the rest was free. The kernels mmap algorithm was predictable: it<BR>> + would start by mapping files at low addresses and work up from there.<BR>> + </PARA><BR>> +<BR>> + <PARA><BR>> + The development of a series of new low level patches violated many of these assumptions, and<BR>> + resulted in Wine needing to force the Win32 address space layout upon the system. This<BR>> + section looks at why and how this is done.<BR>> + </PARA><BR>> +<BR>> + <PARA><BR>> + The exec-shield patch increases security by randomizing the kernels mmap algorithms. Rather<BR>> + than consistently choosing the same addresses given the same sequence of requests, the kernel<BR>> + will now choose randomized addresses. Because the Linux dynamic linker (ld-linux.so.2) loads<BR>> + DSOs into memory by using mmap, this means that DSOs are no longer loaded at predictable<BR>> + addresses, so making it harder to attack software by using buffer overflows. It also attempts<BR>> + to relocate certain binaries into a special low area of memory known as the ASCII armor so<BR>> + making it harder to jump into them when using string based attacks.<BR>> + </PARA><BR>> +<BR>> + <PARA><BR>> + Prelink is a technology that enhances startup times by precalculating ELF global offset<BR>> + tables then saving the results inside the native binaries themselves. By grid fitting each<BR>> + DSO into the address space, the dynamic linker does not have to perform as many relocations<BR>> + so allowing applications that heavily rely on dynamic linkage to be loaded into memory much<BR>> + quicker. Complex C++ applications such as Mozilla, OpenOffice and KDE can especially benefit<BR>> + from this technique.<BR>> + </PARA><BR>> +<BR>> + <PARA><BR>> + The 4G VM split patch was developed by Ingo Molnar. It gives the Linux kernel its own address<BR>> + space, thereby allowing processes to access the maximum addressable amount of memory on a<BR>> + 32-bit machine: 4 gigabytes. It allows people with lots of RAM to fully utilise that in any<BR>> + given process at the cost of performance: as mentioned previously the reason behind giving<BR>> + the kernel a part of each processes address space was to avoid the overhead of switching on<BR>> + each syscall.<BR>> + </PARA><BR>> +<BR>> + <PARA><BR>> + Each of these changes alter the address space in a way incompatible with Windows. Prelink and<BR>> + exec-shield mean that the libraries Wine uses can be placed at any point in the address<BR>> + space: typically this meant that a library was sitting in the region that the EXE you wanted<BR>> + to run had to be loaded (remember that unlike DLLs, EXE files cannot be moved around in<BR>> + memory). The 4G VM split means that programs could receive pointers to the top gigabyte of<BR>> + address space which some are not prepared for (they may store extra information in the high<BR>> + bits of a pointer, for instance). In particular, in combination with exec-shield this one is<BR>> + especially deadly as it's possible the process heap could be allocated beyond<BR>> + ADDRESS_SPACE_LIMIT which causes Wine initialization to fail. <BR>> + </PARA><BR>> +<BR>> + <PARA><BR>> + The solution to these problems is for Wine to reserve particular parts of the address space<BR>> + so that areas that we don't want the system to use will be avoided. We later on<BR>> + (re/de)allocate those areas as needed. One problem is that some of these mappings are put in<BR>> + place automatically by the dynamic linker: for instance any libraries that Wine<BR>> + is linked to (like libc, libwine, libpthread etc) will be mapped into memory before Wine even<BR>> + gets control. In order to solve that, Wine overrides the default ELF initialization sequence<BR>> + at a low level and reserves the needed areas by using direct syscalls into the kernel (ie<BR>> + without linking against any other code to do it) before restarting the standard<BR>> + initialization and letting the dynamic linker continue. This is referred to as the<BR>> + preloader and is found in ld-winepreload.so (loader/preloader.c)<BR>> + </PARA><BR>> +<BR>> + <PARA><BR>> + Once the usual ELF boot sequence has been completed, some native libraries may well have been<BR>> + mapped above the 3gig limit: however, this doesn't matter as 3G is a Windows limit, not a<BR>> + Linux limit. We still have to prevent the system from allocating anything else above there<BR>> + (like the heap or other DLLs) though so Wine performs a binary search over the upper gig of<BR>> + address space in order to iteratively fill in the holes with MAP_NORESERVE mappings so the<BR>> + address space is allocated but the memory to actually back it is not. This code can be found<BR>> + in libs/wine/mmap.c:reserve_area.<BR>> + </PARA><BR>> + <BR>> + </SECT1><BR>> + <BR>> +</CHAPTER><BR>> --- documentation/wine-devel.sgml<BR>> +++ documentation/wine-devel.sgml<BR>> @@ -13,6 +13,7 @@<BR>> <!entity ddraw SYSTEM "ddraw.sgml"><BR>> <!entity multimedia SYSTEM "multimedia.sgml"><BR>> <!entity threading SYSTEM "threading.sgml"><BR>> +<!entity address-space SYSTEM "address-space.sgml"><BR>> <BR>> <!entity implementation SYSTEM "implementation.sgml"><BR>> <!entity porting SYSTEM "porting.sgml"><BR>> @@ -138,6 +139,7 @@<BR>> &implementation;<BR>> &porting;<BR>> &consoles;<BR>> + &address-space; <BR>> &cvs-regression;<BR>> </PART><BR>> <BR>> <BR>> <BR>> </BLOCKQUOTE>