script to run all good tests / all flaky tests / etc.
Dan Kegel
dank at kegel.com
Tue Sep 6 02:24:24 CDT 2011
The buildbot does three things to get fast, reliable test runs:
- it runs a 'safe' set of tests in the background, without DISPLAY
set, in a different wineprefix; this saves three minutes
- it has a blacklist of known flaky or broken tests, listing how it's
broken and linking to the bug report
- it kills wineserver after the run (since some tests leave processes around)
I hoisted these out of the buildbot config and into a separate script,
http://code.google.com/p/winezeug/source/browse/trunk/buildbot/dotests.sh
As a bonus, you can ask it to run all the flaky tests
instead of all the good ones, which might be useful
if you're trolling for test bugs to fix.
Here's the current blacklist from that script:
# Blacklist format
# test condition [bug]
# Tests may appear multiple times in the list.
# Supported conditions:
# FLAKY - fails sometimes
# CRASHY - crashes sometimes
# SYS - always fails on some systems, not on others
# HEAP - fails or crashes if warn+heap
# NOTTY - test fails if output redirected
# BAD64 - always fails on 64 bits
cat > /tmp/blacklist.txt <<_EOF_
dlls/kernel32/tests/heap.ok HEAP
dlls/dsound/tests/ds3d8.ok SYS 8668
dlls/dsound/tests/ds3d.ok SYS 8668
dlls/dsound/tests/dsound8.ok SYS 8668
dlls/dsound/tests/dsound.ok SYS 8668
dlls/user32/tests/input.ok SYS 12053
dlls/user32/tests/msg.ok SYS 12053
dlls/user32/tests/win.ok SYS 12053
dlls/oleaut32/tests/tmarshal.ok BAD64 26768
dlls/ieframe/tests/ie.ok BAD64 26768
dlls/ws2_32/tests/sock.ok CRASHY 28012
dlls/wininet/tests/urlcache.ok FLAKY 28038
dlls/winmm/tests/wave.ok SYS 28048
dlls/winmm/tests/mci.ok SYS 28071
dlls/urlmon/tests/url.ok FLAKY 28108
dlls/winmm/tests/capture.ok SYS 28109
dlls/shell32/tests/shlfolder.ok SYS 28216
dlls/kernel32/tests/process.ok NOTTY 28220
dlls/d3dx9_36/tests/shader.ok HEAP 28255
dlls/kernel32/tests/pipe.ok HEAP 28257
programs/cmd/tests/batch.ok HEAP 28258
dlls/dsound/tests/ds3d8.ok HEAP 28260
dlls/dsound/tests/ds3d.ok HEAP 28260
dlls/mshtml/tests/style.ok HEAP 28262
dlls/winhttp/tests/winhttp.ok SYS 28267
programs/wscript/tests/run.ok BAD64 28285
dlls/msctf/tests/inputprocessor.ok FLAKY 28288
_EOF_
More information about the wine-devel
mailing list