Threading docs [2]

Mike Hearn mh at codeweavers.com
Mon May 10 16:33:18 CDT 2004


Dimi pointed out on IRC that we had dropped the author
for each section system, so here it is again using the 
<author> tag this time.

Mike Hearn <mh at codeweavers.com>
Add some documentation on how threading is implemented in Wine. 
Covers basic Win32 threading primitives, syslevels, the TEB and
pthreads vs kthreads.

--- documentation/wine-devel.sgml
+++ documentation/wine-devel.sgml
@@ -12,6 +12,7 @@
 <!entity opengl SYSTEM "opengl.sgml">
 <!entity ddraw SYSTEM "ddraw.sgml">
 <!entity multimedia SYSTEM "multimedia.sgml">
+<!entity threading SYSTEM "threading.sgml">
 
 <!entity implementation SYSTEM "implementation.sgml">
 <!entity porting SYSTEM "porting.sgml">
@@ -56,6 +57,10 @@
 	<surname>den Haan</surname>
       </author>
       <author>
+        <firstname>Mike</firstname>
+        <surname>Hearn</surname>
+      </author>
+      <author>
 	<firstname>Ove</firstname>
 	<surname>Kaaven</surname>
       </author>
@@ -125,6 +130,7 @@
     &opengl;
     &ddraw;
     &multimedia;
+    &threading;
   </part>
 
   <part id="part-three">
--- /dev/null	2003-09-15 14:40:47.000000000 +0100
+++ documentation/threading.sgml	2004-05-10 22:30:42.000000000 +0100
@@ -0,0 +1,267 @@
+<chapter id="threading">
+  <title>Multi-threading in Wine</title>
+  
+  <para>
+    This section will assume you understand the basics of multithreading. If not there are plenty of
+    good tutorials available on the net to get you started.
+  </para>
+
+  <para>
+    Threading in Wine is somewhat complex due to several factors. The first is the advanced level of
+    multithreading support provided by Windows - there are far more threading related constructs available
+    in Win32 than the Linux equivalent (pthreads). The second is the need to be able to map Win32 threads
+    to native Linux threads which provides us with benefits like having the kernel schedule them without
+    our intervention. While it's possible to implement threading entirely without kernel support, doing so
+    is not desirable on most platforms that Wine runs on.
+  </para>
+
+  <sect1>
+    <title> Threading support in Win32 </title>
+
+    <para>
+      Win32 is an unusually thread friendly API. Not only is in entirely thread safe, but it provides
+      many different facilities to working with threads. These range from the basics such as starting
+      and stopping threads, to the extremely complex such as injecting threads into other processes and
+      COM inter-thread marshalling.
+    </para>
+
+    <para>
+      One of the primary challenges of writing Wine code therefore is ensuring that all our DLLs are
+      thread safe, free of race conditions and so on. This isn't simple - don't be afraid to ask if
+      you aren't sure whether a piece of code is thread safe or not!
+    </para>
+
+    <para>
+      Win32 provides many different ways you can make your code thread safe however the most common
+      are the <emphasis>critical section</emphasis> and the <emphasis>interlocked functions</emphasis>.
+      Critical sections are a type of mutex designed to protect a geographic area of code. If you don't
+      want multiple threads running in a piece of code at once, you can protect them with calls to
+      EnterCriticalSection and LeaveCriticalSection. The first call to EnterCriticalSection by a thread
+      will lock the section and continue without stopping. If another thread calls it then it will block
+      until the original thread calls LeaveCriticalSection again.
+    </para>
+
+    <para>
+      It is therefore vitally important that if you use critical sections to make some code thread-safe,
+      that you check every possible codepath out of the code to ensure that any held sections are left.
+      Code like this:
+    </para>
+
+    <programlisting> if (res != ERROR_SUCCESS) return res;  </programlisting>
+
+    <para>
+      is extremely suspect in a function that also contains a call to EnterCriticalSection. Be careful.
+    </para>
+
+    <para>
+      If a thread blocks while waiting for another thread to leave a critical section, you will
+      see an error from the RtlpWaitForCriticalSection function, along with a note of which
+      thread is holding the lock. This only appears after a certain timeout, normally a few
+      seconds. It's possible the thread holding the lock is just being really slow which is why
+      Wine won't terminate the app like a non-checked build of Windows would, but the most
+      common cause is that for some reason a thread forgot to call LeaveCriticalSection, or died
+      while holding the lock (perhaps because it was in turn waiting for another lock). This
+      doesn't just happen in Wine code: a deadlock while waiting for a critical section could
+      be due to a bug in the app triggered by a slight difference in the emulation.
+    </para>
+    
+    <para>
+      Another popular mechanism available is the use of functions like InterlockedIncrement and
+      InterlockedExchange. These make use of native CPU abilities to execute a single
+      instruction while ensuring any other processors on the system cannot access memory, and
+      allow you to do common operations like add/remove/check a variable in thread-safe code
+      without holding a mutex. These are useful for reference counting especially in
+      free-threaded (thread safe) COM objects.
+    </para>
+
+    <para>
+      Finally, the usage of TLS slots are also popular. TLS stands for thread-local storage, and is
+      a set of slots scoped local to a thread which you can store pointers in. Look on MSDN for the
+      TlsAlloc function to learn more about the Win32 implementation of this. Essentially, the
+      contents of a given slot will be different in each thread, so you can use this to store data
+      that is only meaningful in the context of a single thread. On recent versions of Linux the
+      __thread keyword provides a convenient interface to this functionality - a more portable API
+      is exposed in the pthread library. However, these facilities is not used by Wine, rather, we
+      implement Win32 TLS entirely ourselves.
+    </para>
+  </sect1>
+
+  <sect1>
+    <title> SysLevels </title>
+
+    <para>
+      SysLevels are an undocumented Windows-internal thread-safety system. They are basically
+      critical sections which must be taken in a particular order. The mechanism is generic but
+      there are always three syslevels: level 1 is the Win16 mutex, level 2 is the USER mutex
+      and level 3 is the GDI mutex. Therefore by implication to modify a GDI object you must
+      also be holding the USER lock.
+    </para>
+
+    <para>
+      When entering a syslevel, the code (in dlls/kernel/syslevel.c) will check that a
+      higher syslevel is not already held and produce an error if so. This is because it's not
+      legal to enter level 2 while holding level 3 - first, you must leave level 3.
+    </para>
+
+    <para>
+      Throughout the code you may see calls to _ConfirmSysLevel() and _CheckNotSysLevel(). These
+      functions are essentially assertions about the syslevel states and can be used to check
+      that the rules have not been accidentally violated. In particular, _CheckNotSysLevel()
+      will break (probably into the debugger) if the check fails. If this happens the solution
+      is to get a backtrace and find out, by reading the source of the wine functions called
+      along the way, how Wine got into the invalid state.
+    </para>
+    
+  </sect1>
+
+  <sect1>
+    <title> POSIX threading vs kernel threading </title>
+
+    <para>
+      Wine runs in one of two modes: either pthreads (posix threading) or kthreads (kernel
+      threading). This section explains the differences between them. The one that is used is
+      automatically selected on startup by a small test program which then execs the correct
+      binary, either wine-kthread or wine-pthread. On NPTL-enabled systems pthreads will be
+      used, and on older non-NPTL systems kthreads is selected.
+    </para>
+
+    <para>
+      Let's start with a bit of history. Back in the dark ages when Wines threading support was
+      first implemented a problem was faced - Windows had much more capable threading APIs than
+      Linux did. This presented a problem - Wine works either by reimplementing an API entirely
+      or by mapping it onto the underlying systems equivalent. How could Win32 threading be
+      implemented using a library which did not have all the neeed features? The answer, of
+      course, was that it couldn't be.
+    </para>
+
+    <para>
+      On Linux the pthreads interface is used to start, stop and control threads. The pthreads
+      library in turn is based on top of so-called "kernel threads" which are created using the
+      clone(2) syscall. Pthreads provides a nicer (more portable) interface to this
+      functionality and also provides APIs for controlling mutexes. There is a
+      <ulink url="http://www.llnl.gov/computing/tutorials/pthreads/">
+        good tutorial on pthreads </ulink> available if you want to learn more.
+    </para>
+
+    <para>
+      As pthreads did not provide the necessary semantics to implement Win32 threading, the
+      decision was made to implement Win32 threading on top of the underlying kernel threads by
+      using syscalls like clone directly. This provided maximum flexibility and allowed a
+      correct implementation but caused some bad side effects. Most notably, all the userland
+      Linux APIs assumed that the user was utilising the pthreads library. Some only enabled
+      thread safety when they detected that pthreads was in use - this is true of glibc, for
+      instance. Worse, pthreads and pure kernel threads had strange interactions when run in
+      the same process yet some libraries used by Wine used pthreads internally. Throw in
+      source code porting using WineLib - where you have both UNIX and Win32 code in the same
+      process - and chaos was the result.
+    </para>
+
+    <para>
+      The solution was simple yet ingenius: Wine would provide its own implementation of the pthread
+      library <emphasis>inside</emphasis> its own binary. Due to the semantics of ELF symbol
+      scoping, this would cause Wines own implementations to override any implementation loaded
+      later on (like the real libpthread.so). Therefore, any calls to the pthread APIs in
+      external libraries would be linked to Wines instead of the systems pthreads library, and
+      Wine implemented pthreads by using the standard Windows threading APIs it in turn
+      implemented itself.
+    </para>
+
+    <para>
+      As a result, libraries that only became thread-safe in the presence of a loaded pthreads
+      implementation would now do so, and any external code that used pthreads would actually
+      end up creating Win32 threads that Wine was aware of and controlled. This worked quite
+      nicely for a long time, even though it required doing some extremely un-kosher things like
+      overriding internal libc structures and functions. That is, it worked until NPTL was
+      developed at which point the underlying thread implementation on Linux changed
+      dramatically.
+    </para>
+
+    <para>
+      The fake pthread implementation can be found in loader/kthread.c, which is used to
+      produce to wine-kthread binary. In contrast, loader/pthread.c produces the wine-pthread
+      binary which is used on newer NPTL systems.
+    </para>
+
+    <para>
+      NPTL is a new threading subsystem for Linux that hugely improves its performance and
+      flexibility. By allowing threads to become much more scalable and adding new pthread
+      APIs, NPTL made Linux competitive with Windows in the multi-threaded world. Unfortunately
+      it also broke many assumptions made by Wine (as well as other applications such as the
+      Sun JVM and RealPlayer) in the process.
+    </para>
+
+    <para>
+      There was, however, some good news. NPTL made Linux threading powerful enough
+      that Win32 threads could now be implemented on top of pthreads like any other normal
+      application. There would no longer be problems with mixing win32-kthreads and pthreads
+      created by external libraries, and no need to override glibc internals. As you can see
+      from the relative sizes of the loader/kthread.c and loader/pthread.c files, the
+      difference in code complexity is considerable. NPTL also made several other semantic
+      changes to things such as signal delivery so changes were required in many different
+      places in Wine.
+    </para>
+
+    <para>
+      On non-Linux systems the threading interface is typically not powerful enough to
+      replicate the semantics Win32 applications expect and so kthreads with the
+      pthread overrides are used.
+    </para>
+  </sect1>
+
+  <sect1>
+    <title> The Win32 thread environment </title>
+
+    <para>
+      All Win32 code, whether from a native EXE/DLL or in Wine itself, expects certain constructs to
+      be present in its environment. This section explores what those constructs are and how Wine
+      sets them up. The lack of this environment is one thing that makes it hard to use Wine code
+      directly from standard Linux applications - in order to interact with Win32 code a thread
+      must first be "adopted" by Wine.
+    </para>
+
+    <para>
+      The first thing Win32 code requires is the <emphasis>TEB</emphasis> or "Thread Environment
+      Block". This is an internal (undocumented) Windows structure associated with every thread
+      which stores a variety of things such as TLS slots, a pointer to the threads message queue,
+      the last error code and so on. You can see the definition of the TEB in include/thread.h, or
+      at least what we know of it so far. Being internal and subject to change, the layout of the
+      TEB has had to be reverse engineered from scratch.
+    </para>
+
+    <para>
+      A pointer to the TEB is stored in the %fs register and can be accessed using NtCurrentTeb()
+      from within Wine code. %fs actually stores a selector, and setting it therefore requires
+      modifying the processes local descriptor table (LDT) - the code to do this is in lib/wine/ldt.c.
+    </para>
+
+    <para>
+      The TEB is required by nearly all Win32 code run in the Wine environment, as any wineserver
+      RPC will use it, which in turn implies that any code which could possibly block (for instance
+      by using a critical section) needs it. The TEB also holds the SEH exception handler chain as
+      the first element, so if when disassembling you see code like this:
+    </para>
+
+    <programlisting> movl %esp, %fs:0 </programlisting>
+
+    <para>
+      ... then you are seeing the program set up an SEH handler frame. All threads must have at
+      least one SEH entry, which normally points to the backstop handler which is ultimately
+      responsible for popping up the all-too-familiar "This program has performed an illegal
+      operation and will be terminated" message. On Wine we just drop straight into the debugger.
+      A full description of SEH is out of the scope of this section, however there are some good
+      articles in MSJ if you are interested.
+    </para>
+
+    <para>
+      All Win32-aware threads must have a wineserver connection. Many different APIs
+      require the ability to communicate with the wineserver. In turn, the wineserver must be aware
+      of Win32 threads in order to be able to accurately report information to other parts of the
+      program and do things like route inter-thread messages, dispatch APCs (asynchronous procedure
+      calls) and so on. Therefore a part of thread initialization is initializing the thread
+      serverside. The result is not only correct information in the server, but a set of file
+      descriptors the thread can use to communicate with the server - the request fd, reply fd and
+      wait fd (used for blocking).
+    </para>
+    
+  </sect1>
+</chapter>





More information about the wine-patches mailing list