WWN Issue 74

This is the 74th release of the Wine's kernel cousin publication. Its main goal is to distribute widely what's going on around Wine (the Un*x Windows emulator).

Errata

Ove Kaaven pointed out that <kcref>last week article</kcref> contained errors. Functions were optimized out because of the static attribute (not the unused) and that Alexandre patch only removed this very static attribute. Thanks for pointing out the mistakes.

This week, 91 posts consumed 367 K. There were 34 different contributors. 14 (41%) posted more than once. 17 (50%) posted last week too.

The top 5 posters of the week were:

9 posts in 30K by Jon
8 posts in 44K by Eric Pouech
8 posts in 81K by Francois Gouget
7 posts in 18K by Andreas Mohr
7 posts in 20K by Alexandre Julliard

Wrc and alignment	11 Dec 2000 00:00:00 -0800	Archive
Moritz Barsnick, while trying compiling Wine on Sparc/Solaris, got a segmentation while wrc was compiling a resource file. Ulrich Weigand summarized the list of known issues when using wrc on non Intel CPUs: Wrc is broken here; it thinks it need to byte-swap although the message compiler already wrote the file in big-endian. Thus, it gets nonsense length values and crashes subsequently. Wrc is further broken in that is does not cope with unaligned data found in binary input files or raw data (e.g. icons etc.), and hence causes bus errors on Sparc. Winebuild is also broken in that it always generates i386 assembly code for import thunks and constructor/destructor stubs. Various parts of the main source are also not alignment safe. Ulrich also said he had fixed a few of those issues, and will shortly provide them. The others issues brought up by Moritz are more compiler portability fixes (like generating the correct macros for the system, and reimplementing when needed the inline i386 assembly). Anyway, there's still lots of work to be done.

Wrc and alignment

11 Dec 2000 00:00:00 -0800

GDI lock and 16 bit printer drivers	12 Dec 2000 00:00:00 -0800	Archive
Andreas Mohr reported a deadlock. Basically, his program created a device context. With the new GDI object locking in place, the GDI lock is acquired when creating the object, and will be released after the DC has been created and initialized. During the initialization phase, the driver is called. When the driver is a 16 bit one, the Win16 lock needs to be entered, potentially providing trouble since locks shall be acquired in the other way (the Win16 lock must be acquired before GDI lock, for example). Andreas asked what to do. Alexandre Julliard quickly answered: I'm afraid there is no good solution, short of redefining the GDI driver interface to not call the DC functions with the GDI lock held. It may be easier to forget about running Win16 printer drivers... (does anybody really use this anyway?) Andi quite didn't like the answer: You don't really want to propose this, do you ? ;) So every time we've got an incredibly stupid locking problem, we are supposed to give up major functionality, just like... poooooff.. that's it ! ? I'm pretty sure that several people still use Win16 drivers, as PSDRV support is still a bit weak (I never got it to run so far, but didn't try too hard either). So would there be a "solid" solution, other than just giving up on Win16 drivers completely ? Or in other words: what would need to be done in order to make it work again ? Malte Cornils reminded he had posted about a similar issue, but he didn't get any reply at that time. Alexandre Julliard explained that the 'incredibly stupid locking problem', as Andi put it, comes from that Win16 code is not thread-safe; go complain to Microsoft. If the choice is between a thread-safe GDI and Win16 printer drivers, I think most people will agree GDI is more important, but ended up saying to Andi redefine the whole GDI driver interface. Not terribly complicated, but it's still a lot of work. Have fun ;-) So Andi may (he didn't say he would) rewrite part of GDI driver interface so that it's called without the GDI lock acquired. Stay tuned!

Archive

Francois Gouget submitted a new tool designed to gather information about the functions implemented in the various DLLs, and get a rough idea of what still needs to be done: I analyzed the functions exported by each Windows Dll for Win95, Win98 and Win2000 and compared them with what is found in the spec files. Here's the executive summary:

Set 1	Set 2	Removed / Missing in 2	Added/Extra in 2
95	98	8	982
98	2000	509	2813
95	wine	847	1805
98	wine	1326	1397
2000	wine	2835	517
all	wine	3000	266

So it seems that there are quite a few differences. The last line tells us that there seems to be 3000 functions missing in our spec files but that we export 266 functions that are not present in any of the Windows versions (or it could be that we are missing 2734 functions and have 266 that differ, or something in between).

I'll try to collect WinME and NT4 soon, I'll post an updated diff and executive summary if there is interest or it turns very different results)

So it seems that Wine doesn't cover all the API (even from Windows 95 and 98), but also started to implement part of Windows NT and 2000 features. On the other hand, this approach is partial: the tool can't tell the difference from an empty function (just returning an error code), from a fully implemented function. But also, don't forget that lots of Windows API are only used by very focused applications. Today, efforts are targeted to the areas which are needed by most of the applications (priorization is made according to the usage). Wine will surely not try to be exhaustive in this area.

Feature: Of Dynamic loading in Wine by Eric Pouech

Feature: Of Dynamic loading in Wine by Eric Pouech
<h3>Introduction</h3> Wine recently had lots of new features in the way it handles dynamic linking. Before diving into the details, let's have a quick reminder of Windows and Unix implementation of dynamic linking (DL). Dynamic linking allows to separate executable code between several modules (each stored in a different file), but loaded in memory to create a process image. The DL features, in both Unix and Windows, can be used in two different ways: automatic: when you create a new module (excutable or DL library), you describe at the link stage the DL libraries you depend upon. In that case, you can use direct call to those libraries (as they were directly linked to your code). When your new module is loaded into memory before being executed, the dependencies will be resolved and the other libraries loaded (or almost, see below about delay loading). by hand: in the code of your program, you need to explicitly load DL library `foo` , and then get the address of function `bar` . Using that function pointer, you can then invoke the function. This second method requires to write more code, but is required when you need to load a DL library (or invoke a function) whose name is not known at compilation time. The 'automatic' method is in fact implemented on top of the 'by hand' method, but is magically done by the linker and other compilation chain tools. Windows implement DL in DLLs (dynamic link libraries). Those DLLs can either contain a 16-bit or a 32-bit library. In both cases, the DLL file contains the following features: the code for the implemented functions a list of exported functions (functions which can be called from outside the library) ; those functions are also known as entry points a list of imported functions: a DLL can also use, in turn, another DLL (the first DLL then has to list the used DLL and the used functions from each DLL). To be more precise this feature of imported functions is not specific to DLL but is also shared by executables. (For the technology addicts, the previous feature of exported functions can also be implemented in executable, even if this is less known) a set of resources (images, dialog boxes, menus...) specific entry points (they differ from 16 and 32 bit DLL) to be called upon DLL loading and unloading (allowing for example, allocation or initialization of specific data) Windows usually stores DLL into files with .DLL extensions (some other extensions are used in some cases (like .DRV, .ACM...), but this is out of the scope of this article). Lots of Unices also implement DL features; a DL library is stored into a .so file. Basically, the same type of information as the one for Windows is also needed (code of the library, imported and exported functions, init (resp. fini) functions called when .so file is loaded (unloaded). Of course, the embedding of resources is not included in the .so format. Binary formats of .DLL and .so files are different. In Unix, the name of the .so file is derived from the name of the module (for example, `foo` will be stored in `libfoo.so` file). Unix .so files however have a nice feature, which MS doesn't have. You can have several versions of the same .so (libfoo.so.1.0, libfoo.so.1.2...). When an executable looks for library `foo` , the DL loader first looks for the requested version, like 1.2. If not found, then, it will look for a 1 version, then for a version-less. Microsoft doesn't allow this directly (however, MS started it with names like MFC32.DLL MFC44.DLL, but it's up to the program to search for MFC44.DLL, and if it fails, ask to search for MFC33.DLLn thus not allowing for 'automatic' linking). <h3>Implementation in Wine</h3> Wine implements the various builtin DLLs in a set of .so files. A single .so file can contain several DLLs. Usually, the DLLs are put together because of their strong relationship in the code. A good example is the 16/32 DLLs pair always stored in the same .so file. But, this is not the only case for putting DLLs together. The loader doesn't use the automatic dependencies on those .so files, but rather rely on the dependencies in the DLLs for this matter. When the .so file is loaded, the embedded headers (for 16 and 32 bit DLLs) are then registered into Wine, so that it behaves as if the embedded DLLs had been loaded one by one thru standard Windows mechanisms. When .so file is removed from memory, the DLLs are also unregistered from Wine. The process to build a DLL in Wine is rather simple. All the .spec files are first passed thru a Wine specific tool (called winebuild) which generates C code (and some inline assembly). This code, once compiled, generates the headers' description in a binary form compatible with Windows' implementation. It also contains the init (resp fini) functions. The link process (using standard tools) then generates a .so file. At this level, this .so file doesn't have dependencies on other .so files also embedding DLL headers. (It may still have dependencies on other Wine .so files, like system .so files - libc... - or Wine specific DLLs - like Unicode basics). On the loader side, when DLL `foo` is requested to be loaded as builtin, the loader tries to load the `libfoo.so` file. However, as we've seen, several headers are stored inside a .so file. For example, in `SHELL` and `SHELL32` are stored in `libshell32.so` . How does Wine handle the multiple names? Wine simply creates for all DLLs embedded inside a .so file a symbolic link to the real .so file. Per .so file, Wine requests one DLL to be the owner of the .so file. The resulting .so file is then named after the owner DLL's name (in our example `libshell32.so` ). The build mechanism is now clear: generate the .so file from all the code (its name is derived from the owner name): for all the other DLLs (not the owner) in the .so file, creates a symbolic link from `libnot_owner .so` to `libowner .so` (in our example, a symbolic link from `libshell.so` to `libshell32.so` is created). <h3>Historical view</h3> The process described above can be seen as Step 3 of Wine implementation of DLLs. Let's review the previous steps: Step 1 each DLL already had a .spec file, but the Wine internal headers were not in sync with Windows one. the .spec files were used to generate the relay code (which allows to display the trace with `-debugmsg +relay` ), but also the thunking code for 16 bit DLL (all entry points of 16 DLLs are implemented as 32 functions, so there's a need for some stubs to call between 16 and 32 bit code) all code was stored in a monolithic way (a single executable) Step 2 the previous monolithic binary is now split in several .so files. Each .so file contains one or several headers (16 or 32 bits). the headers are still in a Wine only way. .so files are loaded on demand, but dependencies between DLLs are handled at the .so level, not a the .DLL level Step 3 dependencies between DLLs is now done at the DLL level, as Windows does The process of code modification from step 2 to Step 3 has been dubbed "DLL separation". <h3>Future directions</h3> Most of Wine DLLs has been moved to Step 3 in Wine implementation process. However, a few DLLs are not yet at a proper Step 3 (because they rely on functions internal to the Wine code, located in another DLL, but not exported by this DLL). This is still allowed at Step 2 the .so files mechanism permits it (every function in a .so file is by default exported), but no longer works at Step 3 because the exported functions are listed. This is currently being worked on. Unix .so file implements by default a delay loading mechanism. If DL lib A depends on DL lib B, the Windows standard mechanism implies that B is loaded when A is loaded. Unix .so file mechanism loads DL lib B when the first function from B is invoked. Windows also implements this type of mechanism (known as DELAYLOAD). Wine doesn't support this yet, but it is worked upon (some patches already circulate). Another possible feature would be to keep on merging the two worlds for debugging information. .so files (as ELF modules) embed the debugging information (stabs format). PE headers also embed their debugging information (but each MS compiler has its own format). WineDbg (the Wine own debugger) allows loading debugging information from stabs and PE headers. For example, when the libshell32.so is loaded, WineDbg loads the debugging information from the stabs, but no information from the embedded DLLs. On the other hand, if the builtin `SHELL32` is loaded (and has debugging information), WineDbg loads debugging information from it. However, regular Windows debugger, even if they could run and attach processes under Wine, cannot load information from .so files. The idea (already seen long ago, and possibly implemented at Corel's) would be to translate, in the .so file build process, the stabs format into the MS one, so that Windows debuggers can read and understand it.

<h3>Introduction</h3> Wine recently had lots of new features in the way it handles dynamic linking. Before diving into the details, let's have a quick reminder of Windows and Unix implementation of dynamic linking (DL).

Dynamic linking allows to separate executable code between several modules (each stored in a different file), but loaded in memory to create a process image.

The DL features, in both Unix and Windows, can be used in two different ways:

automatic: when you create a new module (excutable or DL library), you describe at the link stage the DL libraries you depend upon. In that case, you can use direct call to those libraries (as they were directly linked to your code). When your new module is loaded into memory before being executed, the dependencies will be resolved and the other libraries loaded (or almost, see below about delay loading).
by hand: in the code of your program, you need to explicitly load DL library foo , and then get the address of function bar . Using that function pointer, you can then invoke the function. This second method requires to write more code, but is required when you need to load a DL library (or invoke a function) whose name is not known at compilation time. The 'automatic' method is in fact implemented on top of the 'by hand' method, but is magically done by the linker and other compilation chain tools.

Windows implement DL in DLLs (dynamic link libraries). Those DLLs can either contain a 16-bit or a 32-bit library. In both cases, the DLL file contains the following features:

the code for the implemented functions
a list of exported functions (functions which can be called from outside the library) ; those functions are also known as entry points
a list of imported functions: a DLL can also use, in turn, another DLL (the first DLL then has to list the used DLL and the used functions from each DLL). To be more precise this feature of imported functions is not specific to DLL but is also shared by executables. (For the technology addicts, the previous feature of exported functions can also be implemented in executable, even if this is less known)
a set of resources (images, dialog boxes, menus...)
specific entry points (they differ from 16 and 32 bit DLL) to be called upon DLL loading and unloading (allowing for example, allocation or initialization of specific data)

Windows usually stores DLL into files with .DLL extensions (some other extensions are used in some cases (like .DRV, .ACM...), but this is out of the scope of this article).

Lots of Unices also implement DL features; a DL library is stored into a .so file. Basically, the same type of information as the one for Windows is also needed (code of the library, imported and exported functions, init (resp. fini) functions called when .so file is loaded (unloaded). Of course, the embedding of resources is not included in the .so format. Binary formats of .DLL and .so files are different.

In Unix, the name of the .so file is derived from the name of the module (for example, foo will be stored in libfoo.so file). Unix .so files however have a nice feature, which MS doesn't have. You can have several versions of the same .so (libfoo.so.1.0, libfoo.so.1.2...). When an executable looks for library foo , the DL loader first looks for the requested version, like 1.2. If not found, then, it will look for a 1 version, then for a version-less. Microsoft doesn't allow this directly (however, MS started it with names like MFC32.DLL MFC44.DLL, but it's up to the program to search for MFC44.DLL, and if it fails, ask to search for MFC33.DLLn thus not allowing for 'automatic' linking).

<h3>Implementation in Wine</h3>

Wine implements the various builtin DLLs in a set of .so files. A single .so file can contain several DLLs. Usually, the DLLs are put together because of their strong relationship in the code. A good example is the 16/32 DLLs pair always stored in the same .so file. But, this is not the only case for putting DLLs together.

The loader doesn't use the automatic dependencies on those .so files, but rather rely on the dependencies in the DLLs for this matter. When the .so file is loaded, the embedded headers (for 16 and 32 bit DLLs) are then registered into Wine, so that it behaves as if the embedded DLLs had been loaded one by one thru standard Windows mechanisms.

When .so file is removed from memory, the DLLs are also unregistered from Wine.

The process to build a DLL in Wine is rather simple. All the .spec files are first passed thru a Wine specific tool (called winebuild) which generates C code (and some inline assembly). This code, once compiled, generates the headers' description in a binary form compatible with Windows' implementation. It also contains the init (resp fini) functions.

The link process (using standard tools) then generates a .so file. At this level, this .so file doesn't have dependencies on other .so files also embedding DLL headers. (It may still have dependencies on other Wine .so files, like system .so files - libc... - or Wine specific DLLs - like Unicode basics).

On the loader side, when DLL foo is requested to be loaded as builtin, the loader tries to load the libfoo.so file. However, as we've seen, several headers are stored inside a .so file. For example, in SHELL and SHELL32 are stored in libshell32.so . How does Wine handle the multiple names? Wine simply creates for all DLLs embedded inside a .so file a symbolic link to the real .so file. Per .so file, Wine requests one DLL to be the owner of the .so file. The resulting .so file is then named after the owner DLL's name (in our example libshell32.so ). The build mechanism is now clear: generate the .so file from all the code (its name is derived from the owner name): for all the other DLLs (not the owner) in the .so file, creates a symbolic link from

libnot_owner
.so

libowner
.so

(in our example, a symbolic link from libshell.so to libshell32.so is created).

<h3>Historical view</h3>

The process described above can be seen as Step 3 of Wine implementation of DLLs. Let's review the previous steps:

Step 1
- each DLL already had a .spec file, but the Wine internal headers were not in sync with Windows one.
- the .spec files were used to generate the relay code (which allows to display the trace with -debugmsg +relay ), but also the thunking code for 16 bit DLL (all entry points of 16 DLLs are implemented as 32 functions, so there's a need for some stubs to call between 16 and 32 bit code)
- all code was stored in a monolithic way (a single executable)
Step 2
- the previous monolithic binary is now split in several .so files. Each .so file contains one or several headers (16 or 32 bits).
- the headers are still in a Wine only way.
- .so files are loaded on demand, but dependencies between DLLs are handled at the .so level, not a the .DLL level
Step 3
- dependencies between DLLs is now done at the DLL level, as Windows does

The process of code modification from step 2 to Step 3 has been dubbed "DLL separation".

<h3>Future directions</h3>

Most of Wine DLLs has been moved to Step 3 in Wine implementation process. However, a few DLLs are not yet at a proper Step 3 (because they rely on functions internal to the Wine code, located in another DLL, but not exported by this DLL). This is still allowed at Step 2 the .so files mechanism permits it (every function in a .so file is by default exported), but no longer works at Step 3 because the exported functions are listed. This is currently being worked on.

Unix .so file implements by default a delay loading mechanism. If DL lib A depends on DL lib B, the Windows standard mechanism implies that B is loaded when A is loaded. Unix .so file mechanism loads DL lib B when the first function from B is invoked. Windows also implements this type of mechanism (known as DELAYLOAD). Wine doesn't support this yet, but it is worked upon (some patches already circulate).

Another possible feature would be to keep on merging the two worlds for debugging information. .so files (as ELF modules) embed the debugging information (stabs format). PE headers also embed their debugging information (but each MS compiler has its own format). WineDbg (the Wine own debugger) allows loading debugging information from stabs and PE headers. For example, when the libshell32.so is loaded, WineDbg loads the debugging information from the stabs, but no information from the embedded DLLs. On the other hand, if the builtin SHELL32 is loaded (and has debugging information), WineDbg loads debugging information from it. However, regular Windows debugger, even if they could run and attach processes under Wine, cannot load information from .so files. The idea (already seen long ago, and possibly implemented at Corel's) would be to translate, in the .so file build process, the stabs format into the MS one, so that Windows debuggers can read and understand it.

World Wine News