wined3d: fix check for P8 render target

[Alexander Dorofeyev]

Hi.

Yes, exactly, the AVP1 game is using p8 textures and color keys on some of them
for transparency.

The thing is, the "index_in_alpha" hack DOES get enabled on its P8 3D textures,
and that's the problem I'm trying to fix. I don't see the "primary render target
is p8" check in the code, instead it looks like it checks that render_targets[0]
is a render target.

In patch I proposed I check that the passed surface is a render target, and,
since it's passed to d3dfmt_p8_init_palette then I guess it means there's a p8
render target and in such case the index_in_alpha hack is enabled. Although I
don't check anything beyond just that it's a render target, I guess it could be
some offscreen buffer. Is that the problem, essentially?

Regarding GTA - I could try looking for demo for tests, but can you clarify - is
it really GTA4? GTA4 is upcoming game AFAIK, maybe you mean GTA3?

Roderick Colenbrander wrote:
> 
> Hi,
> 
> Unfortunately this patch is not correct and I haven't looked exactly what the real issue is.
> 
> In short most hardware (except if it has GL_EXT_paletted_texture) doesn't support 8bit textures and most of the time we do 8bit->32bit conversion in software. On hardware that supports fragment shaders we perform the conversion using a shader and this is a lot faster (it is 'free'). The palette is uploaded in d3dfmt_p8_init_palette. 
> 
> So the p8 conversion is done in hardware but in some cases we need to read back the data from the card. The p8 -> 32bit conversion was done behind the back of the game, so the game expects 8bit data back. A palette contains upto 256 entries and each color can appear multiple times in the palette. So the inverse palette lookup doesn't have a 1:1 mapping and this is problematic.
> 
> The trick we do is to store the 8bit color index in the alpha component of the palette and use this to read back the correct index during the 32bit->8bit conversion. The problem is that most games need alpha and for that reason we only use this 'hack' when the primary render target is using 8bit. In this case you can't be using 3D and only old apps like StarCraft use this.
> 
> Your game is likely using some p8 textures and is likely suffering from the 32 -> 8bit problem. I haven't check AVP but if this is the real issue there it can't be fixed. It would break other games like GTA4 and others.
> 
> Regards,
> Roderick Colenbrander