Flaky tests are starting to get real annoying...
dank at kegel.com
Tue Aug 12 12:58:47 CDT 2008
So, one of the things one learns when writing a
patch robot is that flaky tests are very annoying.
Each time it gets a new git tree,
the robot does five baseline "make -k test" runs,
remembers the tests that fail, and doesn't penalize
patches for failing any of those tests. See
Annoyingly, that's not enough. Some tests
stubbornly refuse to fail during the baseline test
runs. So I added a second, manual blacklist
for those tests; see
The list is currently
and will continue growing as I keep plugging away at getting the
patch robot happy.
Is anybody else seeing this kind of flakiness?
(If you're not, try running patchwatcher for a while :-)
FWIW, I'm running the tests on hardy with a fresh metacity
(as described in http://wiki.winehq.org/MakeTestFailures ).
More information about the wine-devel