[Bug 48164] test.winehq.org should provide an efficient way to detect new failures

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48164

François Gouget <fgouget at codeweavers.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #1 from François Gouget <fgouget at codeweavers.com> ---
This is done.

Problem
-------

1. Detecting when a new failure slips past the TestBot.

    https://test.winehq.org/data/patterns.html

 -> The new page has one pattern per test with failures (including those with
missing dlls, etc). As planned the failure count is color coded so changes are
easily visible.
 -> All the patterns are on a single page which is easier to review than
visiting one page per test.
 -> The tests are put in four categories, with the ones most likely to contain
new failures coming first. This helps reduce the number of tests that need to
be looked at to have a good chance of detecting the new failures.


2. Detecting when the results on a VM degrade.

 -> At the top of the page a color coded pattern shows the number of failed
test units for each VM. Changes in color make result degradation obvious. This
also allows detecting when a VM stops reporting results (whether because it has
too many failures, times out or other reasons).


3. Comparing the results of two machines of different platforms.

 -> The pattern at the top of the page contains results for all platforms. The
results are sorted by platform to make it easy to compare their result as a
group, but the results of different platform are just a few lines apart.


4. Show only the failures, not all the lines with skips and todos.

 -> This is the normal behavior of the new pattern page.


5. Have a page showing only the TestBot results.

 -> Two pages show the results for the Windows and Wine TestBot VMs
respectively.

    https://test.winehq.org/data/patterns-tb-win.html
    https://test.winehq.org/data/patterns-tb-wine.html


Implementation tweaks
---------------------

* Instead of a single color gradient there is a chain of color gradients to
help with patterns that have many different failure counts. This is
configurable and currently goes like this: dark cyan -> green -> yellow -> red.

* For each test the page links to related commits. For each commit it show if
it patched the test itself, a shared test resource or the Wine module. This
simplifies reviewing potential culprits when a test starts failing.

* For each test the page links to the related bugs. This simplifies checking if
someone has already analyzed the bug or tried fixing it. If the bug has a
regression commit id that commit is shown as such.

* The page also shows bugs for tests that have no failure. This simplifies
verifying that the bugs have been updated when a test is fixed.


Commits
-------

There are about 27 commits spread between the two below:

commit bce49667e6418c326ade5704ee65c5bf8c6930d6
Author: Francois Gouget <fgouget at codeweavers.com>
Date:   Tue Apr 27 12:45:07 2021 +0200

    winetest/build-patterns: Add a page showing failure patterns.

    The new page shows the failure patterns across time and test
    configurations to help detect Wine changes that cause new failures,
    and identify when failures happen in specific configurations.

    Wine-Bug: https://bugs.winehq.org/show_bug.cgi?id=48164
    Signed-off-by: Francois Gouget <fgouget at codeweavers.com>
    Signed-off-by: Alexandre Julliard <julliard at winehq.org>


commit 0ef48b83ac84f3a59ed5521b24d5a7a8d1dcde17
Author: Francois Gouget <fgouget at codeweavers.com>
Date:   Wed May 19 03:15:13 2021 +0200

    winetest/build-patterns: Show Potentially obsolete bugs.

    Bugs related to a test that has no failure are likely to be out of
    date, except if they are related to memory issues.

    Signed-off-by: Francois Gouget <fgouget at codeweavers.com>
    Signed-off-by: Alexandre Julliard <julliard at winehq.org>

-- 
Do not reply to this email, post in Bugzilla using the
above URL to reply.
You are receiving this mail because:
You are watching all bug changes.