[tools 0/6] Known failures tracking
fgouget at codeweavers.com
Wed Jun 15 11:21:21 CDT 2022
This patchset provides support for tracking known failures in the
TestBot results. The main goal is to avoid false positives but there
are some side benefits as well:
* The job reports sent to developers link to the relevant WineHQ bugs
which gives them more exposure and may help get them fixed faster.
* The failure details pages will eventually link to all the tasks where
the failure happened, which could be valuable to find patterns.
* The failure page tracks when it was last seen. This simplifies
spotting bugs which are fixed and could be closed.
(Or failures where the regular expression needs to be updated / fixed)
The patchset mostly follows the scheme described in bug 48912:
There are some differences though:
* The bug envisionned this scheme purely as a blacklist to avoid false
positives. The implementation can certainly be used in this way but
it is not limited to failures that cause false positives.
Tracking other failures can provide the side benefits mentionned
above. The only limit is the maintenance work required to track
* As a consequence the FailureBlacklist class was renamed to Failure,
and the FailureBlacklistUses was renamed to TaskFailures since it
links Tasks with the related Failures.
* The FailureBlacklistVMs purpose is not really described in the bug but
I think the goal was to provide a way to deckare that the failure is
expected to only happen on specific test configurations.
Instead the Failures class has a ConfigRegExp regular expression
which specifies which test configurations the failure applies to.
The test 'configuration name' is of the form 'VM:logfile' which
allows matching specific Windows versions (thanks to the TestBot VM
naming scheme), specific Windows locales (part of the VM name),
specific bitnesses (part of the log filename), or specific Wine
locales (part of the log filename). For instance:
^w8 -> all Windows 8 VMs
:exe -> all Windows tests (exe32.report and exe64.report)
^w10.*_ja.*: -> all Windows 10 Japanese test configurations
win|wow -> all Wine test configurations (win32.report, ...)
(win|wow).*_fr -> French Wine test configurations
* Changes in FailureBlacklists -> Failures
- Primary key: (Bug, TestModule, TestUnit) -> Id
Some issues generate quite a few failures. So in case trying to
match them all results in a regular expression that's too long
(either because of the database limit or for readability), this
change makes it possible to add multiple Failure objects to track
- TestModule -> ErrorGroup
The .errors files split the errors in groups. Each group
corresponds to a test module, but extra errors (too much output,
missing summary lines, etc) are in a separate group. So I renamed
TestModule to ErrorGroup as a reminder to the coder that this is
meant to match the .errors group names, not just test modules.
- Name -> (BugStatus, BugDescription)
Instead of having a name describing the failure, this patchset uses
the corresponding WineHQ bug description.
The TestBot also retrieves the bug status which allows checking
that closed bugs dont happen anymore, or open bugs that should be
- LastUsed -> (LastNew, LastOld)
The LastUse field was split into LastNew and LastOld to track
whether the failure would cause false positives without the known
* Changes in FailureBlacklistUses -> TaskFailures
- (Bug, TestModule, TestUnit) -> FailureId
This is a consequence of the Failure primary key change described
- () -> (TaskLog)
This allows tracking which task log the failure was found in. This
is quite important for Wine tasks since they run all the test
configurations in just one task.
- () -> (NewCount, OldCount)
This provides a count of matching failures and whether they would
have been tagged as old or new without this patchset.
This provides a quick way to know whether there is the same number
of failures every time or whether the number of failures changes
from one run to the next.
It also provides a quick way to know how many failures to look for
when jumping to the log.
Francois Gouget (6):
testbot: Add tables to track the test failures.
testbot/web: Add pages to add, edit and show known failures.
testbot: Match the failures with the test logs.
testbot/web: Link job details to the related known failures.
testbot/WineSendLog: Take into account the known failures.
testbot/UpdateFailures: Add a script to refresh the failures' bug
testbot/bin/UpdateFailures.pl | 192 ++++++
testbot/bin/WineSendLog.pl | 78 ++-
testbot/ddl/update47.sql | 33 +
testbot/ddl/winetestbot.sql | 32 +
testbot/doc/INSTALL.txt | 5 +
testbot/doc/winetestbot-schema.dia | 852 +++++++++++++++++++++---
testbot/lib/WineTestBot/CGI/PageBase.pm | 1 +
testbot/lib/WineTestBot/Config.pm | 6 +-
testbot/lib/WineTestBot/Failures.pm | 184 +++++
testbot/lib/WineTestBot/Jobs.pm | 6 +
testbot/lib/WineTestBot/LogUtils.pm | 180 ++++-
testbot/lib/WineTestBot/TaskFailures.pm | 120 ++++
testbot/lib/WineTestBot/Tasks.pm | 4 +
testbot/web/FailureDetails.pl | 119 ++++
testbot/web/FailuresList.pl | 160 +++++
testbot/web/JobDetails.pl | 54 +-
testbot/web/WineTestBot.css | 3 +
17 files changed, 1907 insertions(+), 122 deletions(-)
create mode 100755 testbot/bin/UpdateFailures.pl
create mode 100644 testbot/ddl/update47.sql
create mode 100644 testbot/lib/WineTestBot/Failures.pm
create mode 100644 testbot/lib/WineTestBot/TaskFailures.pm
create mode 100644 testbot/web/FailureDetails.pl
create mode 100644 testbot/web/FailuresList.pl
More information about the wine-devel