script to run all good tests / all flaky tests / etc.

Dan Kegel dank at
Tue Sep 6 02:24:24 CDT 2011

The buildbot does three things to get fast, reliable test runs:
- it runs a 'safe' set of tests in the background, without DISPLAY
set, in a different wineprefix; this saves three minutes
- it has a blacklist of known flaky or broken tests, listing how it's
broken and linking to the bug report
- it kills wineserver after the run (since some tests leave processes around)

I hoisted these out of the buildbot config and into a separate script,

As a bonus, you can ask it to run all the flaky tests
instead of all the good ones, which might be useful
if you're trolling for test bugs to fix.

Here's the current blacklist from that script:

# Blacklist format
# test condition [bug]
# Tests may appear multiple times in the list.
# Supported conditions:
# FLAKY - fails sometimes
# CRASHY - crashes sometimes
# SYS - always fails on some systems, not on others
# HEAP - fails or crashes if warn+heap
# NOTTY - test fails if output redirected
# BAD64 - always fails on 64 bits

cat > /tmp/blacklist.txt <<_EOF_
    dlls/kernel32/tests/heap.ok     HEAP
    dlls/dsound/tests/ds3d8.ok      SYS    8668
    dlls/dsound/tests/ds3d.ok       SYS    8668
    dlls/dsound/tests/dsound8.ok    SYS    8668
    dlls/dsound/tests/dsound.ok     SYS    8668
    dlls/user32/tests/input.ok      SYS    12053
    dlls/user32/tests/msg.ok        SYS    12053
    dlls/user32/tests/win.ok        SYS    12053
    dlls/oleaut32/tests/tmarshal.ok BAD64  26768
    dlls/ieframe/tests/ie.ok        BAD64  26768
    dlls/ws2_32/tests/sock.ok       CRASHY 28012
    dlls/wininet/tests/urlcache.ok  FLAKY  28038
    dlls/winmm/tests/wave.ok        SYS    28048
    dlls/winmm/tests/mci.ok         SYS    28071
    dlls/urlmon/tests/url.ok        FLAKY  28108
    dlls/winmm/tests/capture.ok     SYS    28109
    dlls/shell32/tests/shlfolder.ok SYS    28216
    dlls/kernel32/tests/process.ok  NOTTY  28220
    dlls/d3dx9_36/tests/shader.ok   HEAP   28255
    dlls/kernel32/tests/pipe.ok     HEAP   28257
    programs/cmd/tests/batch.ok     HEAP   28258
    dlls/dsound/tests/ds3d8.ok      HEAP   28260
    dlls/dsound/tests/ds3d.ok       HEAP   28260
    dlls/mshtml/tests/style.ok      HEAP   28262
    dlls/winhttp/tests/winhttp.ok   SYS    28267
    programs/wscript/tests/run.ok   BAD64  28285
    dlls/msctf/tests/inputprocessor.ok FLAKY 28288

More information about the wine-devel mailing list