Marking test cases as flaky?
nerv at dawncrow.de
Wed Sep 7 07:12:22 CDT 2011
Am 07.09.2011 06:14, schrieb Dan Kegel:
> Hi folks,
> I've been running "make test" a lot lately, and occasionally
> (it happened five times today), a test will fail out of
> the blue, in one out of twenty or a hundred runs.
> It would be bad form for buildbot to reject a patch
> because of a spurious test failure, so I've been making a list of
> tests that fail, adding to it each time I run into a test
> that fails occasionally, and skipping the tests on that list
> during "make test". This is all fine... but one of the tests that
> fails occasionally is ddraw/visual.ok (see
> http://bugs.winehq.org/show_bug.cgi?id=28301 ).
> It really doesn't feel right to skip that entirely just
> because a few particular pixel comparisons fail sometimes.
> Plan A would be to fix the tests to not be flaky, but that's hard,
> and the buildbot needs a quick solution.
> Plan B might be to conditionally compile the flaky tests away, e.g.:
Plan C could be to do what testbot does and remember possible fails, maybe including the out of the blue ones.
Best Regards, André Hentschel
More information about the wine-devel