Henri Verbeet : wined3d: Properly compare integers in wined3d_pipeline_layout_vk_compare().

[Alexandre Julliard]

Module: wine
Branch: master
Commit: 9b98d86f292891e55ab33e830470f281ce9a10ae
URL:    https://source.winehq.org/git/wine.git/?a=commit;h=9b98d86f292891e55ab33e830470f281ce9a10ae

Author: Henri Verbeet <hverbeet at codeweavers.com>
Date:   Thu Jan 20 17:30:56 2022 +0100

wined3d: Properly compare integers in wined3d_pipeline_layout_vk_compare().

As pointed out by Giovanni, modular subtraction doesn't produce a total order;
in particular, it's not transitive. For the values likely to be encountered by
wined3d_pipeline_layout_vk_compare(), this is perhaps unlikely to be an issue
in practice. However, that's not necessarily the case for e.g. handles or
masks.

Signed-off-by: Henri Verbeet <hverbeet at codeweavers.com>
Signed-off-by: Alexandre Julliard <julliard at winehq.org>

---

 dlls/wined3d/context_vk.c      | 5 +++--
 dlls/wined3d/wined3d_private.h | 5 +++++
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/dlls/wined3d/context_vk.c b/dlls/wined3d/context_vk.c
index fd6d810f0e2..cf6247387ca 100644
--- a/dlls/wined3d/context_vk.c
+++ b/dlls/wined3d/context_vk.c
@@ -1797,9 +1797,10 @@ static int wined3d_pipeline_layout_vk_compare(const void *key, const struct wine
     const struct wined3d_pipeline_layout_key_vk *a = key;
     const struct wined3d_pipeline_layout_key_vk *b = &WINE_RB_ENTRY_VALUE(entry,
             const struct wined3d_pipeline_layout_vk, entry)->key;
+    int ret;
 
-    if (a->binding_count != b->binding_count)
-        return a->binding_count - b->binding_count;
+    if ((ret = wined3d_uint32_compare(a->binding_count, b->binding_count)))
+        return ret;
     return memcmp(a->bindings, b->bindings, a->binding_count * sizeof(*a->bindings));
 }
 
diff --git a/dlls/wined3d/wined3d_private.h b/dlls/wined3d/wined3d_private.h
index 5fa51a844ac..c9d83b3fbc7 100644
--- a/dlls/wined3d/wined3d_private.h
+++ b/dlls/wined3d/wined3d_private.h
@@ -447,6 +447,11 @@ static inline unsigned int wined3d_popcount(unsigned int x)
 #endif
 }
 
+static inline int wined3d_uint32_compare(uint32_t x, uint32_t y)
+{
+    return (x > y) - (x < y);
+}
+
 #define ORM_BACKBUFFER  0
 #define ORM_FBO         1