Marking test cases as flaky?

[Dan Kegel]

Hi folks,
I've been running "make test" a lot lately, and occasionally
(it happened five times today), a test will fail out of
the blue, in one out of twenty or a hundred runs.
It would be bad form for buildbot to reject a patch
because of a spurious test failure, so I've been making a list of
tests that fail, adding to it each time I run into a test
that fails occasionally, and skipping the tests on that list
during "make test".   This is all fine... but one of the tests that
fails occasionally is ddraw/visual.ok (see
http://bugs.winehq.org/show_bug.cgi?id=28301 ).
It really doesn't feel right to skip that entirely just
because a few particular pixel comparisons fail sometimes.

Plan A would be to fix the tests to not be flaky, but that's hard,
and the buildbot needs a quick solution.

Plan B might be to conditionally compile the flaky tests away, e.g.:

--- a/dlls/ddraw/tests/visual.c
+++ b/dlls/ddraw/tests/visual.c
@@ -1490,6 +1490,7 @@ static void D3D1_TextureMapBlendTest(void)
         ok(hr == D3D_OK, "IDirect3DDevice3_EndScene failed, hr = %08x\n", hr);
     }

+#ifndef WINE_SKIP_FLAKY_TEST_SEE_BUG_28301
     color = D3D1_getPixelColor(DirectDraw1, Surface1, 5, 5);
     red =   (color & 0x00ff0000) >> 16;
     green = (color & 0x0000ff00) >>  8;
@@ -1513,6 +1514,7 @@ static void D3D1_TextureMapBlendTest(void)
     green = (color & 0x0000ff00) >>  8;
     blue =  (color & 0x000000ff);
     ok(red == 0 &&  green == 0 && blue == 0xff, "Got color %08x,
expected 000000ff or near\n", color);
+#endif

Then my blacklist would mostly be a bunch of -D definitions in CFLAGS,
one per flaky test.
Or we could use a single define for all flaky tests.

I may try that privately, to increase buildbot's test coverage.

Would something like that be useful for other people, or
should I just stick with private patches?
- Dan