Marking test cases as flaky?

[André Hentschel]

Am 07.09.2011 06:14, schrieb Dan Kegel:
> Hi folks,
> I've been running "make test" a lot lately, and occasionally
> (it happened five times today), a test will fail out of
> the blue, in one out of twenty or a hundred runs.
> It would be bad form for buildbot to reject a patch
> because of a spurious test failure, so I've been making a list of
> tests that fail, adding to it each time I run into a test
> that fails occasionally, and skipping the tests on that list
> during "make test".   This is all fine... but one of the tests that
> fails occasionally is ddraw/visual.ok (see
> http://bugs.winehq.org/show_bug.cgi?id=28301 ).
> It really doesn't feel right to skip that entirely just
> because a few particular pixel comparisons fail sometimes.
> 
> Plan A would be to fix the tests to not be flaky, but that's hard,
> and the buildbot needs a quick solution.
> 
> Plan B might be to conditionally compile the flaky tests away, e.g.:
> 

  Plan C could be to do what testbot does and remember possible fails, maybe including the out of the blue ones.

-- 

Best Regards, André Hentschel