RFC How to get rid of "always new" TestBot false positives?

[Henri Verbeet]

On Fri, 27 Mar 2020 at 16:24, Francois Gouget <fgouget at codeweavers.com> wrote:
> So I'm resubmitting this patch hoping for some feedback since the last
> time it just dropped off the mailing list [1].
>
> The issue is that some test failures include values that change on every
> run which prevents the TestBot from knowing whether they are new or not.
> In turn this means the TestBot systematically blames the author of
> whichever patch is being tested for these failures.
>
If the main goal is to stop the testbot from being ignored, and to
limit the number of new failures sneaking in, would it make sense to
start with something fairly blunt, like ignoring failures for tests on
unreliable configurations? E.g., suppose ddraw:ddraw7 reliably passed
on w1064v1507, but not w1064v1809, you'd then blacklist all of
ddraw:ddraw7 on w1064v1809. That means you potentially ignore some
ddraw:ddraw7 tests that are reliable, but it would still be an
improvement over effectively ignoring everything. Perhaps more
importantly, it also means you can incrementally work on reducing the
size of the blacklist, instead of having to do that work upfront.