"make test" redux 2

[Dan Kegel]

Wine's test suite is now nine years old, and thanks in large part
to Ge's testbot, new tests tend to actually work on windows.
There's a lot of passing tests at http://test.winehq.org, which is awesome.
Wine developers routinely use it to avoid sending bad patches.

And yet, some bad patches do get sent.
Running all the incoming patches from wine-patches though the
reliable subset of the tests automatically has been finding
about one real bug a day.  Sometimes I can warn the author
soon enough that he can send a revised patch before the bad one
is committed.  Automated testing ftw!

But sometimes bad patches don't fail during the few test runs
currently done, get committed, and then start causing unrelated
patches to fail randomly, poisoning future test runs until it the bug
is fixed or the bad patch reverted.
It would be nice if Alexandre reverted bad patches as soon as
the buildbot started going red, without waiting for his usual
daily batch of commits, and without waiting for someone to
fix the bad patch.  (Maybe this can be discussed at wineconf.)

Right, so that's the story for keeping new bugs out of the
tree.  And then there are all the old bugs.  Getting them
fixed sometime would be nice; until then, the buildbot just
skips known bad tests.

Here are the tests that fail or crash sometimes for me when run
normally on 32 bit systems.
(This list is a lot bigger than the one I posted back in 2009, mostly
because I'm running the tests a lot more, and noticing ones that
only fail occasionally.)

http://bugs.winehq.org/show_bug.cgi?id=28301 dlls/ddraw/tests/visual.ok
http://bugs.winehq.org/show_bug.cgi?id=28257 dlls/kernel32/tests/pipe.ok
http://bugs.winehq.org/show_bug.cgi?id=28220 dlls/kernel32/tests/process.ok
http://bugs.winehq.org/show_bug.cgi?id=28295 dlls/mshtml/tests/htmldoc.ok
http://bugs.winehq.org/show_bug.cgi?id=28290 dlls/shell32/tests/shelllink.ok
http://bugs.winehq.org/show_bug.cgi?id=28323 dlls/urlmon/tests/protocol.ok
http://bugs.winehq.org/show_bug.cgi?id=28108 dlls/urlmon/tests/url.ok
http://bugs.winehq.org/show_bug.cgi?id=20149 dlls/user32/tests/static.ok
http://bugs.winehq.org/show_bug.cgi?id=28038 dlls/wininet/tests/urlcache.ok
http://bugs.winehq.org/show_bug.cgi?id=28102 dlls/ws2_32/tests/sock.ok

And another handful that fail or crash sometimes for me when run with
WINEDEBUG=warn+heap:

http://bugs.winehq.org/show_bug.cgi?id=28260 dlls/dsound/tests/ds3d8.ok
http://bugs.winehq.org/show_bug.cgi?id=28260 dlls/dsound/tests/ds3d.ok
http://bugs.winehq.org/show_bug.cgi?id=28316 dlls/kernel32/tests/debugger.ok
http://bugs.winehq.org/show_bug.cgi?id=28257 dlls/kernel32/tests/pipe.ok
http://bugs.winehq.org/show_bug.cgi?id=28262 dlls/mshtml/tests/style.ok
http://bugs.winehq.org/show_bug.cgi?id=28258 programs/cmd/tests/batch.ok

There are also tests I don't run on the buildbot because they only pass
on other systems with particular window managers or sound hardware or software:

http://bugs.winehq.org/show_bug.cgi?id=8668 dlls/dsound/tests/ds3d8.ok
http://bugs.winehq.org/show_bug.cgi?id=8668 dlls/dsound/tests/ds3d.ok
http://bugs.winehq.org/show_bug.cgi?id=8668 dlls/dsound/tests/dsound8.ok
http://bugs.winehq.org/show_bug.cgi?id=8668 dlls/dsound/tests/dsound.ok
http://bugs.winehq.org/show_bug.cgi?id=28216 dlls/shell32/tests/shlfolder.ok
http://bugs.winehq.org/show_bug.cgi?id=12053 dlls/user32/tests/input.ok
http://bugs.winehq.org/show_bug.cgi?id=12053 dlls/user32/tests/msg.ok
http://bugs.winehq.org/show_bug.cgi?id=12053 dlls/user32/tests/win.ok
http://bugs.winehq.org/show_bug.cgi?id=28267 dlls/winhttp/tests/winhttp.ok
http://bugs.winehq.org/show_bug.cgi?id=28109 dlls/winmm/tests/capture.ok
http://bugs.winehq.org/show_bug.cgi?id=28071 dlls/winmm/tests/mci.ok
http://bugs.winehq.org/show_bug.cgi?id=28048 dlls/winmm/tests/wave.ok

More info at http://wiki.winehq.org/BuildBot
- Dan