WineTest failure patterns preview

[Zebediah Figura (she/her)]

On 4/23/21 7:54 AM, Francois Gouget wrote:
> 
> I did not get to send this patch set yet so here's a preview:
> 
> * Windows test patterns
>    http://fgouget.free.fr/tmp/winepatterns/patterns-tb-win.html
> 
> * Wine test patterns
>    http://fgouget.free.fr/tmp/winepatterns/patterns-tb-wine.html
> 
> So for each test unit that had failures you have:
> * one line per test configuration (report tag)
> * one row per WineTest build (daily commit)
> 
> Each cell has a one character description of the test result (with more
> details in the tooltip) and is color coded to help identify patterns.
> Each 'failure' cell also links to the corresponding report page.
> 
> 
> The main goal of these pages for me is to simplify detecting new
> failures in the WineTest results... and not just in cases where there
> was no failure before since there are so many cases with preexisting
> failures already. In the patterns an increase in the number of failures
> shows up as a color change which makes them easy to spot.
> 
> A secondary goal is to allow comparing the before/after results when a
> test configuration (VM or otherwise) is modified.
> 
> A side effect is that these can also help identify the different failure
> modes. For instance comctl32:monthcal fails on wednesdays, except in
> Korean where it's all the time:
> http://fgouget.free.fr/tmp/winepatterns/patterns-tb-win.html#comctl32:monthcal
> 
> 
> To help with these tasks the page also sorts the test units to show
> those that look like they have new failures first. So starting the
> review from the top is most likely to point to the new issues.
> 
> 

Looks like a more sophisticated version of 
<https://www.winehq.org/~jwhite/2deb8c2825af.html>, which is definitely 
a nice resource when I'm trying to put effort into fixing test failures.

I guess the tests are color-coded by number of failures, modulo some 
constant? I like the idea. I will note though that some of those colours 
seem hard to tell apart, e.g. the shades of green in wine d3d9:device. 
Also I guess they aren't consistent across tests for some reason?