Exception Handling with a "bad" ESP

[Glenn Wurster]

I've been doing some debugging of the exception handler and have come
across a problem for which I'm not sure what the best solution is.  It
involves an application using the ESP for something other than the
stack.

The current DIB code uses protection to detect when the application
modifies the DIB bits and respond accordingly.  It does this by
disabling read and/or write access to pages and then catching
exceptions when they are thrown so that it can keep the X11 and
windows versions of the DIB seporate but reflect changes appropriately
(There may be other pieces of Wine which use similar tactics).

The application I am debugging, however, uses the ESP as a
counter/memory offset in it's updating of the DIB.  While not the best
selection of registers, the application relies on ESP not being used
for anything else while the DIB updates take place (In Windows this
assumption is true).

A made-up example of assembly code is as follows:

  mov esp, [0x12345678]        ; Save the ESP
  mov 0x128796, esp            ; Set up ESP for the DIB modification
  mov 0, ebp
loop:
  mov ebp, [sp + bp]           ; Write out the byte to the DIB
  inc ebp
  sub esp, 0x10000             ; The high word is a counter
  jns loop
  mov [0x12345678], esp        ; Restore the ESP

Now, one of the first things our exception handler in signal_i386.c
does is set up the stack layout to deliver the exception to the
application (see the setup_exception function), but in this case ESP
definately does not point to the stack (and the application never
intended for an exception to occurr).  It instead contains a counter
in the high word, and a memory offset in the low word.  Because of
this, there is a double fault which happens and Wine actually
Segfaults as opposed to breaking into the debugger.

So, what I'm wondering is if anyone who has experiance with the
exception handling code has any thoughts on how this problem can be
fixed.

If your interested in actual application code which does this, Alpha
Centauri is what I found the initial problem with.  You can download
the demo at http://www.firaxis.com/downloads/Demo/smac_demo1_1.exe. The
assembly code in the application which causes the crash is located
starting at around offset 0x0060d954.

As an interesting side-effect, if there are other applications which
are using ESP in a "bad" manor, then we may be corrupting memory
whenever we generate an exception within Wine.  This issue would be
very hard to track down, which makes me think that only exceptions
which would normally be generated in Windows should cause Wine to
modify the application stack.

Comments?  Suggestions?

Glenn.