wined3d: fix check for P8 render target

[Roderick Colenbrander]

> Hi.
> 
> I think this part of d3dfmt_p8_init_palette has a bug, because it appears
> to be
> checking that render target is a render target. Incorrectly going into
> index_in_alpha codepath here causes this bug:
> 
> http://bugs.winehq.org/show_bug.cgi?id=10797
> 
> in Aliens vs Predator 1 game - it prefers P8 textures and sets color key
> on some
> textures for transparency and alpha channel is wrong on such textures in
> game.
> 
> ---
>      dlls/wined3d/surface.c |    7 ++-----
>      1 files changed, 2 insertions(+), 5 deletions(-)
> 
> 

Hi,

Unfortunately this patch is not correct and I haven't looked exactly what the real issue is.

In short most hardware (except if it has GL_EXT_paletted_texture) doesn't support 8bit textures and most of the time we do 8bit->32bit conversion in software. On hardware that supports fragment shaders we perform the conversion using a shader and this is a lot faster (it is 'free'). The palette is uploaded in d3dfmt_p8_init_palette. 

So the p8 conversion is done in hardware but in some cases we need to read back the data from the card. The p8 -> 32bit conversion was done behind the back of the game, so the game expects 8bit data back. A palette contains upto 256 entries and each color can appear multiple times in the palette. So the inverse palette lookup doesn't have a 1:1 mapping and this is problematic.

The trick we do is to store the 8bit color index in the alpha component of the palette and use this to read back the correct index during the 32bit->8bit conversion. The problem is that most games need alpha and for that reason we only use this 'hack' when the primary render target is using 8bit. In this case you can't be using 3D and only old apps like StarCraft use this.

Your game is likely using some p8 textures and is likely suffering from the 32 -> 8bit problem. I haven't check AVP but if this is the real issue there it can't be fixed. It would break other games like GTA4 and others.

Regards,
Roderick Colenbrander
-- 
Psssst! Schon vom neuen GMX MultiMessenger gehört?
Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger?did=10