[PATCH] d2d1: Fix the stack overflow error caused by d2d_cdt_triangulate().

[Changsheng Chen]

Hello, Henri Verbeet. Yes, it is at least greater than or equal to three, but it happens in the upper function d2d_path_geometry_triangulate(). Through some testing methods, I locate it before d2d_path_geometr_triangulate() calls d2d_cdt_triangulate(), it will change the value of vertex_count , Make it less than 3. When d2d_cdt_triangulate() is actually received, the value of vertex_count will cause a stack overflow error.
In d2d_path_geometry_triangulate(), the code block to change the size of vertex_count is as follows:


for (i = 1; i <vertex_count; ++i)
     {
         if (!memcmp(&vertices[i-1], &vertices[i], sizeof(*vertices)))
         {
             --vertex_count;
             memmove(&vertices[i], &vertices[i + 1], (vertex_count-i) * sizeof(*vertices));
             --i;
         }
     }


We can see that the value of vertex_count has been reduced. At the same time, we saw in the test that after passing in d2d_cdt_triangulate(), the vertex_count is less than 3.

 
Of course, under normal circumstances, it is difficult to test and find this problem. I found that it was a drawing application that I migrated. Whenever I draw with the pen, I click the mouse habitually, and the application gets stuck. The debug log throws a stack overflow error. By tracking this problem, I found it and found a solution to it, which is through this patch of mine.

 
 
------------------ Original ------------------
From:  "Henri Verbeet"<hverbeet at gmail.com>;
Date:  Mon, Aug 30, 2021 01:09 PM
To:  "陈长胜"<chenchangsheng at uniontech.com>; 
Cc:  "wine-devel"<wine-devel at winehq.org>; 
Subject:  Re: [PATCH] d2d1: Fix the stack overflow error caused by d2d_cdt_triangulate().

 

On Mon, 30 Aug 2021 at 09:48, 陈长胜 <chenchangsheng at uniontech.com> wrote:
>
> When vertex_count==1 or 0, d2d_cdt_triangulate will always be called by itself, causing stack overflow error.

How does it end up getting called like that, do you have a test?

If I had to guess, I suppose it's possible that we end up with less
than three vertices after eliminating duplicates in
d2d_path_geometry_triangulate(). In that case though, it would make
more sense to move the vertex count check in that function after the
duplicate elimination.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20210831/5c76e8f5/attachment.htm>