[Bug 12652] Rainbow six Vegas 2 fails to launch (number of multisample buffers > 8 causes out-of-bounds crash)

Mon Feb 3 18:43:14 CST 2014

http://bugs.winehq.org/show_bug.cgi?id=12652

Anastasius Focht <focht at gmx.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Keywords|                            |obfuscation
                 CC|                            |focht at gmx.net
          Component|-unknown                    |directx-d3d
            Summary|Rainbow six Vegas 2 Fails   |Rainbow six Vegas 2 fails
                   |to launch                   |to launch (number of
                   |                            |multisample buffers > 8
                   |                            |causes out-of-bounds crash)

--- Comment #39 from Anastasius Focht <focht at gmx.net> ---
Hello folks,

the copy protection (Safedisc v4.x) is not an issue here.

It makes the thing a bit harder to debug because the interesting process is
already a debuggee under control by Safedisc process (debugger).

Installation from original media (CD) is fine.
The game gets automatically updated to latest version (v1.03) via Ubisoft
launcher on first start.

Scan of main executable:

--- snip ---
-=[ ProtectionID v0.6.5.5 OCTOBER]=-
(c) 2003-2013 CDKiLLER & TippeX
Build 31/10/13-21:09:09
Ready...

Scanning -> Z:\home\focht\.wine\drive_c\Program Files\Ubisoft\Tom Clancy's
Rainbow Six Vegas 2\Binaries\R6Vegas2_Game.exe
File Type : 32-Bit Exe (Subsystem : Win GUI / 2), Size : 30445712 (01D09090h)
Byte(s)
-> File Appears to be Digitally Signed @ Offset 01D07E08h, size : 01288h /
04744 byte(s)
-> File has 1449480 (0161E08h) bytes of appended data starting at offset
01BA6000h
[File Heuristics] -> Flag : 00000100000000000100000000000111 (0x04004007)
[Entrypoint Section Entropy] : 6.65
[Debug Info]
Characteristics : 0x0 | TimeDateStamp : 0x484570E1 | MajorVer : 0 / MinorVer :
0 -> (0.0)
Type : 2 -> CodeView | Size : 0x77 (119)
AddressOfRawData : 0x16A67F0 | PointerToRawData : 0x16A67F0
CvSig : 0x53445352 | SigGuid 6885E896-11FF-4207-A354F06C9E4F0EA2
Age : 0x16 | Pdb :
x:\Perforce\bishop\CodePcPortGermany\UnrealEngine3\Binaries\Lib\Retail\PCLaunch-KellerGame.pdb

[!] Safedisc v4.85.000 detected !
[i] Appended data contents....
   [.] o: 0x01BA6028  / t: <0xA8726B03> <0xEF01996C> <0x00000001> / s: 00302967
byte(s) -> ~deaa13.tmp
   [.] o: 0x01BEFFC6  / t: <0xA8726B03> <0xEF01996C> <0x0000044C> / s: 00015887
byte(s) -> clcd32.dll
   [.] o: 0x01BF3DFC  / t: <0xA8726B03> <0xEF01996C> <0x0000044C> / s: 00004122
byte(s) -> clcd16.dll
   [.] o: 0x01BF4E3A  / t: <0xA8726B03> <0xEF01996C> <0x0000044D> / s: 00037971
byte(s) -> mcp.dll
   [.] o: 0x01BFE2B6  / t: <0xA8726B03> <0xEF01996C> <0x0000000B> / s: 00005446
byte(s) -> SecDrv04.VxD
   [.] o: 0x01BFF821  / t: <0xA8726B03> <0xEF01996C> <0x00000000> / s: 00072192
byte(s) -> ~e5.0001
   [.] o: 0x01C11248  / t: <0xA8726B03> <0xEF01996C> <0x00000000> / s: 00045056
byte(s) -> PfdRun.pfd
   [.] o: 0x01C1C270  / t: <0xA8726B03> <0xEF01996C> <0x00000000> / s: 00965148
byte(s) -> ~df394b.tmp
[CompilerDetect] -> Visual C++ 8.0 (Visual Studio 2005)
- Scan Took : 1.618 Second(s) [000000566h tick(s)] [533 scan(s) done]
--- snip ---

The trace log (+relay) doesn't reveal the problem at all, one has to debug it.

The game engine translates various game settings to string representations
This doesn't work out well for the number of multisample buffers available with
modern graphics cards/drivers.

In my case Wine's D3D reports up to 16x/32x to the game engine, depending on
chosen fb config.

--- snip ---
...
0032:trace:d3d:wined3d_init Initializing adapters.
0032:trace:d3d:wined3d_adapter_init adapter 0x1d9448, ordinal 0.
0032:trace:d3d:wined3d_adapter_init Allocated LUID 00000000:00000400 for
adapter 0x1d9448.
0032:trace:d3d:wined3d_caps_gl_ctx_create getting context...
0032:trace:d3d:wined3d_adapter_init_gl_caps adapter 0x1d9448.
0032:trace:d3d:wined3d_adapter_init_gl_caps GL_RENDERER: "GeForce GT
425M/PCIe/SSE2".
0032:trace:d3d:wined3d_adapter_init_gl_caps GL_VENDOR: "NVIDIA Corporation".
0032:trace:d3d:wined3d_adapter_init_gl_caps GL_VERSION: "4.2.0 NVIDIA 304.116".
0032:trace:d3d:wined3d_parse_gl_version Found OpenGL version 4.2. 
...
0032:trace:d3d:wined3d_adapter_init_fb_cfgs iPixelFormat=148,
iPixelType=0x202b, doubleBuffer=0, RGBA=8/8/8/8, depth=24, stencil=8,
samples=16, windowDrawable=1
0032:trace:d3d:wined3d_adapter_init_fb_cfgs iPixelFormat=149,
iPixelType=0x202b, doubleBuffer=1, RGBA=8/8/8/0, depth=24, stencil=0,
samples=32, windowDrawable=1
...
--- snip ---

The game engine code was written ~2007-2008, expecting maximum value 8x
What happens is an out-of-bounds lookup...

--- snip ---
...
114CF8FB  8981 80080000      MOV DWORD PTR DS:[ECX+880],EAX
114CF901  8B15 D0AA3512      MOV EDX,DWORD PTR DS:[1235AAD0]
114CF907  8B82 80080000      MOV EAX,DWORD PTR DS:[EDX+880]  ; MSAA level index
114CF90D  8B0D 08D31B12      MOV ECX,DWORD PTR DS:[121BD308]
114CF913  8B1485 90D31B12    MOV EDX,DWORD PTR DS:[EAX*4+121BD390] ; lookup
114CF91A  A1 0CD31B12        MOV EAX,DWORD PTR DS:[121BD30C]
114CF91F  6A 00              PUSH 0
114CF921  6A 00              PUSH 0
114CF923  51                 PUSH ECX ; UNICODE "R6Menus"
114CF924  52                 PUSH EDX ; MSAA level string
114CF925  50                 PUSH EAX ; UNICODE "SettingsMenu"
114CF926  8D4C24 24          LEA ECX,DWORD PTR SS:[ESP+24]
114CF92A  51                 PUSH ECX
114CF92B  E8 205FADFF        CALL R6Vegas2.10FA5850
114CF930  8BF8               MOV EDI,EAX
114CF932  8B15 08D31B12      MOV EDX,DWORD PTR DS:[121BD308]
114CF938  A1 8CD31B12        MOV EAX,DWORD PTR DS:[121BD38C]
114CF93D  8B0D 0CD31B12      MOV ECX,DWORD PTR DS:[121BD30C]
...
--- snip ---

Stack before call to 0x10FA5850:

--- snip ---
0063CCEC       0063CD14
0063CCF0       11EF2274  UNICODE "SettingsMenu"
0063CCF4       5673676E  ; problem
0063CCF8       11EF2264  UNICODE "R6Menus"
0063CCFC       00000000
0063CD00       00000000
--- snip ---

Stack var 0x0063CCF4 should actually point to a wide-character string (via
lookup table).
The crash happens in a vsnprintf() like function that tries to dereference this
memory location (value is later part of va_list arg array).

Dump of internal string table:

(0x121BD390 = index 0)

[0x10*4+0x121BD390] = 0x121BD3D0 -> 0x5673676E

--- snip ---
...
121BD390    11EF25B0   UNICODE "ScreenMSAALevel0"
121BD394    11EF25D4   UNICODE "ScreenMSAALevel1"
121BD398    11EF25F8   UNICODE "ScreenMSAALevel2"
121BD39C    11EF261C   UNICODE "ScreenMSAALevel3"
121BD3A0    11EF2640   UNICODE "ScreenMSAALevel4"
121BD3A4    11EF2664   UNICODE "ScreenMSAALevel5"
121BD3A8    11EF2688   UNICODE "ScreenMSAALevel6"
121BD3AC    11EF26AC   UNICODE "ScreenMSAALevel7"
121BD3B0    11EF26D0   UNICODE "ScreenMSAALevel8"
121BD3B4    00000000   ....
121BD3B8    11F2FA94
121BD3BC    00000000   ....
121BD3C0    56413F2E   .?AV
121BD3C4    614D3652   R6Ma
121BD3C8    53616D67   gmaS
121BD3CC    69747465   etti
121BD3D0    5673676E   ngsV
121BD3D4    6F656469   ideo
121BD3D8    00004040   @@..
...
--- snip ---

Because the game engine code is broken in that aspect you need to externally
limit the reported number of multisample buffers or disable multisampling (not
really a good option).
For NVIDIA there exist tools to override the settings ... not sure about ATI,
Intel.
As mentioned above, another quick hack is to disable multisample at all using
Wine registry, allowing the game to reach main menu and write out settings
file.

I think it's out of Wine's scope to allow other than enable/disable tweaks via
registry.

IMHO 'WONTFIX' (broken game).

On Windows there most likely exist a specific app shim/driver tweak (= compat
mode) for this game that does the same under the hood, artificially limiting
reported MSAA value.

The Steam version of this game probably has this issue addressed by using a
modified/patched game engine.

Regards

-- 
Do not reply to this email, post in Bugzilla using the
above URL to reply.
You are receiving this mail because:
You are watching all bug changes.