[PATCH 3/4] wined3d: Add a slop for color keying.

[Henri Verbeet]

On 20 August 2015 at 10:03, Stefan Dösinger <stefan at codeweavers.com> wrote:
> @@ -6332,7 +6332,8 @@ static GLuint shader_glsl_generate_ffp_fragment_shader(struct shader_glsl_priv *
>      }
>
>      if (settings->color_key_enabled)
> -        shader_addline(buffer, "if (all(equal(tex0, color_key))) discard;\n");
> +        shader_addline(buffer, "if (all(lessThan(abs(tex0 - color_key), vec4(%s)))) discard;\n",
> +                wined3d_color_key_precision);
>
...
> +/* Normalization of B5G6R5_UNORM textures is horribly imprecise if we don't have
> + * GL_RGB_565 support. 1 / 256 (~0.0039) works in practice, but is awfully close
> + * to the next possible value in 8 bit formats. 1 / 384 is too precise for some
> + * 5 and 6 bit channel values at least on Nvidia.
> + *
> + * An exact comparison isn't reliable for any format, except for the normalized
> + * values 0.0 and 1.0. */
> +const char *wined3d_color_key_precision = "0.003";
>
Shouldn't you just pass the color key as a range to the shader
instead? E.g. in a 6 bpc format anything between 0.484f and 0.500f
would get quantized to 0x1f. (Which, other floating point issues
aside, would correspond to about 0.008 slop here. For a 5 bpc format
you'd even have 0.016.)