TestBot News : Failure tracking

[Francois Gouget]

The TestBot now has a list of known failures which it can identify and 
link to the corresponding Wine bug.

The main benefit is that the TestBot knows your patches cannot be the 
cause of known failures, even if they look new [1]. The list of known 
failures is not yet complete [2] but you can expect that over the next 
few weeks the cases where the TestBot incorrectly claims your patch 
introduces new failures should get much rarer.


Failure tracking also provides you with other benefits:

The task full logs now have an index of the known failures they contain 
to help you quickly go to the first corresponding failure line.
(and a pair of additional entries point you to the first and last 
failure)


Also the failure lines are annotated with the corresponding bug id(s). 
This id is a link that takes you to a page that:
* Shows how the TestBot identified the failure.
* Links to the Wine bug discussing the failure.
* And show all the tasks and logs where the failure occurred. You 
  can use this to quickly access these logs and look for common 
  patterns.


You can also get a list of the known failures by clicking on the 
Failures entry in the TestBot's sidebar.

  https://testbot.winehq.org/FailuresList.pl

For each failure you will see when it last happened and the status of 
the corresponding bug (new, closed, etc). This is a good way to identify 
failures that seem to be fixed.


Finally the job report emails also list the known failures that happened 
while testing your patch. Consider this a reminder that your help in 
fixing them would be greatly appreciated, particularly as you may 
already know the area since you patched it.


[1] The TestBot normally diffs the test run's failures against the past 
    WineTest reports and considers any line marked as new (literally '+' 
    in unified diff parlance), as a new failure.

    This means any failure that's rare enough to not be present in any 
    of the past WineTest runs of the same test configuration, will be 
    flagged as new.

    Failures where the message changes (beyond basic line number 
    changes, addresses in exceptions, etc.) will also look new. This 
    could be because a patch changed the failure message, but more often 
    it is because the message itself contains a pointer, handle, 
    uninitialized data, elapsed time, etc.

[2] The goal is not necessarily to track every test failure but to track 
    all those that can cause false positives. So a test that fails 
    systematically may not be tracked (although if it has a Wine bug it 
    probably should be). But tests that tend to be incorrectly tagged as 
    new due to the issues in [1] should all be.

    The way I plan to get there is to monitor the TestBot's WineTest 
    jobs for a while and add to the known failure list whenever new 
    unknown failures appear there.

-- 
Francois Gouget <fgouget at codeweavers.com>