RFC How to get rid of "always new" TestBot false positives?

[Zebediah Figura]

On 3/27/20 12:14 PM, Henri Verbeet wrote:
> On Fri, 27 Mar 2020 at 16:24, Francois Gouget <fgouget at codeweavers.com> wrote:
>> So I'm resubmitting this patch hoping for some feedback since the last
>> time it just dropped off the mailing list [1].
>>
>> The issue is that some test failures include values that change on every
>> run which prevents the TestBot from knowing whether they are new or not.
>> In turn this means the TestBot systematically blames the author of
>> whichever patch is being tested for these failures.
>>
> If the main goal is to stop the testbot from being ignored, and to
> limit the number of new failures sneaking in, would it make sense to
> start with something fairly blunt, like ignoring failures for tests on
> unreliable configurations? E.g., suppose ddraw:ddraw7 reliably passed
> on w1064v1507, but not w1064v1809, you'd then blacklist all of
> ddraw:ddraw7 on w1064v1809. That means you potentially ignore some
> ddraw:ddraw7 tests that are reliable, but it would still be an
> improvement over effectively ignoring everything. Perhaps more
> importantly, it also means you can incrementally work on reducing the
> size of the blacklist, instead of having to do that work upfront.
> 

For what it's worth, I also think this is a good idea. That way we don't
have to pollute the Wine source, but it's still obvious what tests need
fixing.