[PATCH vkd3d] vkd3d-shader/hlsl: Fix type comparison for column-major matrices.

[Giovanni Mascellani]

Hi,

Il 28/09/21 04:23, Zebediah Figura (she/her) ha scritto:
> (I don't like the situation with matrix majority, or the amount of 
> thought I had to put into this to make sure this was correct... but I 
> suppose I'm at least partly to blame for making it this way.)

Yeah, it's a bit strange, because we try to represent with two bits what 
clearly has precisely two alternative possibilities (column- or row-major).

I think this is an artifact of how parsing works: each keyword simply 
sets the corresponding bit, and then there is some logic that checks 
that it is not the case that both bits are set. It is not really wrong 
to keep this behavior, provided that, once the check is done, the 
row-major bit is always used and the column-major is ignored.

In order to reduce the probability of mistakes, I would drop the 
constant HLSL_MODIFIERS_MAJORITY_MASK (unless it turns out that there 
are cases where it is useful) and always query majority using a 
dedicated macro like

#define HLSL_IS_ROW_MAJOR(modifiers) ((modifiers) & HLSL_MODIFIER_ROW_MAJOR)

Thanks, Giovanni.