[PATCH vkd3d 4/6] vkd3d-shader/hlsl: Fix common shape computation.

[Giovanni Mascellani]

Matrices of shape 1xN always decay to vectors of length N.

Moreover, the assumption about the size of matrices is not correct:
it is legitimate to compose a matrix 2x2 with a vector of length 4,
in which case it appears that the result has the shape of the first
(leftmost) operand. Even for matrices 1xN or Nx1, the result is not
always a vector: in general it has the shape of the first operand
again, which gets possibly decayed to a vector by the rule above.

This algorithm is unfortunately not complete yet: sometimes, but not
always, vectors of length 1 are decayed to scalars.

Signed-off-by: Giovanni Mascellani <gmascellani at codeweavers.com>
---
 libs/vkd3d-shader/hlsl.y | 16 ++++------------
 1 file changed, 4 insertions(+), 12 deletions(-)

diff --git a/libs/vkd3d-shader/hlsl.y b/libs/vkd3d-shader/hlsl.y
index 3119380c..111db8cc 100644
--- a/libs/vkd3d-shader/hlsl.y
+++ b/libs/vkd3d-shader/hlsl.y
@@ -971,18 +971,7 @@ static bool expr_common_shape(struct hlsl_ctx *ctx, struct hlsl_type *t1, struct
     }
     else
     {
-        /* Two vectors or a vector and a matrix (matrix must be 1xn or nx1) */
-        unsigned int max_dim_1, max_dim_2;
-
-        max_dim_1 = max(t1->dimx, t1->dimy);
-        max_dim_2 = max(t2->dimx, t2->dimy);
-        if (t1->dimx * t1->dimy == t2->dimx * t2->dimy)
-        {
-            *type = HLSL_CLASS_VECTOR;
-            *dimx = max(t1->dimx, t2->dimx);
-            *dimy = 1;
-        }
-        else if (max_dim_1 <= max_dim_2)
+        if (t1->dimx * t1->dimy <= t2->dimx * t2->dimy)
         {
             *type = t1->type;
             *dimx = t1->dimx;
@@ -996,6 +985,9 @@ static bool expr_common_shape(struct hlsl_ctx *ctx, struct hlsl_type *t1, struct
         }
     }
 
+    if (*type == HLSL_CLASS_MATRIX && *dimy == 1)
+        *type = HLSL_CLASS_VECTOR;
+
     return true;
 }
 
-- 
2.33.0