[PATCH vkd3d 4/6] vkd3d-shader/hlsl: Fix common shape computation.

[Zebediah Figura]

On 10/14/21 8:37 AM, Giovanni Mascellani wrote:
> Matrices of shape 1xN always decay to vectors of length N.
> 
> Moreover, the assumption about the size of matrices is not correct:
> it is legitimate to compose a matrix 2x2 with a vector of length 4,
> in which case it appears that the result has the shape of the first
> (leftmost) operand. Even for matrices 1xN or Nx1, the result is not
> always a vector: in general it has the shape of the first operand
> again, which gets possibly decayed to a vector by the rule above.

"Fix X" is not usually a very good commit message, and here as often it 
is a sign that multiple things are being done at once.

I.e. I think the "matrices decay to vectors" part makes sense as a 
separate commit (e.g. I guess float4x1 + float4x2 = float4 would be 
fixed by that case).

> 
> This algorithm is unfortunately not complete yet: sometimes, but not
> always, vectors of length 1 are decayed to scalars.
> 
> Signed-off-by: Giovanni Mascellani <gmascellani at codeweavers.com>
> ---
>   libs/vkd3d-shader/hlsl.y | 16 ++++------------
>   1 file changed, 4 insertions(+), 12 deletions(-)
> 
> diff --git a/libs/vkd3d-shader/hlsl.y b/libs/vkd3d-shader/hlsl.y
> index 3119380c..111db8cc 100644
> --- a/libs/vkd3d-shader/hlsl.y
> +++ b/libs/vkd3d-shader/hlsl.y
> @@ -971,18 +971,7 @@ static bool expr_common_shape(struct hlsl_ctx *ctx, struct hlsl_type *t1, struct
>       }
>       else
>       {
> -        /* Two vectors or a vector and a matrix (matrix must be 1xn or nx1) */
> -        unsigned int max_dim_1, max_dim_2;
> -
> -        max_dim_1 = max(t1->dimx, t1->dimy);
> -        max_dim_2 = max(t2->dimx, t2->dimy);
> -        if (t1->dimx * t1->dimy == t2->dimx * t2->dimy)
> -        {
> -            *type = HLSL_CLASS_VECTOR;
> -            *dimx = max(t1->dimx, t2->dimx);
> -            *dimy = 1;
> -        }
> -        else if (max_dim_1 <= max_dim_2)
> +        if (t1->dimx * t1->dimy <= t2->dimx * t2->dimy)
>           {
>               *type = t1->type;
>               *dimx = t1->dimx;
> @@ -996,6 +985,9 @@ static bool expr_common_shape(struct hlsl_ctx *ctx, struct hlsl_type *t1, struct
>           }
>       }
>   
> +    if (*type == HLSL_CLASS_MATRIX && *dimy == 1)
> +        *type = HLSL_CLASS_VECTOR;
> +
>       return true;
>   }
>   
>