[Bug 47998] Better deal with random test failures

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=47998

François Gouget <fgouget at codeweavers.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |FIXED

--- Comment #1 from François Gouget <fgouget at codeweavers.com> ---
This is fixed, including for the deduplication part.
There are a couple caveats though:

* There are failures rare enough they are not present in a given
configuration's WineTest history, but frequent enough that they will regularly
happen on jobs running the test on a dozen or more configurations. The only
solution for those is to fix the test.

* There are a lot of 'always new' failures, that is where the failure message
changes every time thus preventing the TestBot from detecting that they
happened before. Those cause both false positives and also prevent the
deduplication from being effective. The fix is to either remove the variable
part (typically a pointer or handle value) or to enclose it in delimiters that
the TestBot can recognize see bug 48209).


commit 8c6d8e66595d376adbe3cb1f199dbb69331739a8
Author: Francois Gouget <fgouget at codeweavers.com>
Date:   Mon Feb 24 04:45:23 2020 +0100

    testbot/LogUtils: Deduplicate the latest WineTest reports.

    There is no need to keep old logs if they don't contain errors that are
    already present in the latest one. It cuts down on the number of reports
    that test results need to be compared to to detect new failures.

    Signed-off-by: Francois Gouget <fgouget at codeweavers.com>
    Signed-off-by: Alexandre Julliard <julliard at winehq.org>

commit da678a9ebfe3862378543e80b03cd7c5496334cc
Author: Francois Gouget <fgouget at codeweavers.com>
Date:   Tue Feb 18 17:33:09 2020 +0100

    testbot: Take into account multiple WineTest reports.

    Instead of keeping only the latest WineTest results, keep all of them
    and only consider an error new if it does not appears in one of the
    WineTest results. This minimizes the risk of tagging a failure as new
    when it happens intermitently.
    However it also means the number of reference WineTest results can grow
    without bounds so purge them after $JobPurgeDays.
    Since all reference reports end up in the latest/ directory this also
    makes it unnecessary to go fishing for them in the jobs/ directory when
    running UpdateTaskLogs on a specific job. Also filtering reference
    reports to only use those older than the task can be done based on the
    reference report mtime instead of having to keep trust the jobid order.

    Note: UpdateTaskLog can optionally be run to update the new/old status
    in the errors cache files.

    Wine-Bug: https://bugs.winehq.org/show_bug.cgi?id=47998
    Signed-off-by: Francois Gouget <fgouget at codeweavers.com>
    Signed-off-by: Alexandre Julliard <julliard at winehq.org>

-- 
Do not reply to this email, post in Bugzilla using the
above URL to reply.
You are receiving this mail because:
You are watching all bug changes.