[7/10] WineD3D: Enable color keying only for surfaces without an alpha channel

[Stefan Dösinger]

As per nvidia's comments on color keying, color keying is not emulated for 
alpha enabled surfaces
-------------- next part --------------
From 9e4a3380aa260e479d40d9b7a0bb6089041068ea Mon Sep 17 00:00:00 2001
From: Stefan Doesinger <stefan at codeweavers.com>
Date: Sun, 11 Feb 2007 15:16:41 +0100
Subject: [PATCH] WineD3D: Enable color keying only for surfaces without an alpha channel

This is to stay consistent with the surface conversion code. It converts only surfaces without an alpha channel
for color keying.
---
 dlls/wined3d/state.c |   14 ++++++++++++--
 1 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/dlls/wined3d/state.c b/dlls/wined3d/state.c
index 02c80c7..165064d 100644
--- a/dlls/wined3d/state.c
+++ b/dlls/wined3d/state.c
@@ -337,9 +337,19 @@ static void state_alpha(DWORD state, IWineD3DStateBlockImpl *stateblock, WineD3D
      * used WINED3DRS_COLORKEYENABLE state(which is d3d <= 3 only). The texture function will call alpha
      * in case it finds some texture+colorkeyenable combination which needs extra care.
      */
-    if(stateblock->textures[0]) {
+    if(stateblock->textures[0] && stateblock->textureDimensions[0] == GL_TEXTURE_2D) {
         surf = (IWineD3DSurfaceImpl *) ((IWineD3DTextureImpl *)stateblock->textures[0])->surfaces[0];
-        if(surf->CKeyFlags & DDSD_CKSRCBLT) enable_ckey = TRUE;
+
+        if(surf->CKeyFlags & DDSD_CKSRCBLT) {
+            const PixelFormatDesc *fmt = getFormatDescEntry(surf->resource.format);
+            /* The surface conversion does not do color keying conversion for surfaces that have an alpha
+             * channel on their own. Likewise, the alpha test shouldn't be set up for color keying if the
+             * surface has alpha bits
+             */
+            if(fmt->alphaMask == 0x00000000) {
+                enable_ckey = TRUE;
+            }
+        }
     }
 
     if (stateblock->renderState[WINED3DRS_ALPHATESTENABLE] ||
-- 
1.4.4.3