[Bug 9916] "make test" usually fails

wine-bugs at winehq.org wine-bugs at winehq.org
Thu Sep 8 11:27:33 CDT 2011


http://bugs.winehq.org/show_bug.cgi?id=9916

--- Comment #95 from Dan Kegel <dank at kegel.com> 2011-09-08 11:27:31 CDT ---
Wine's tests are in a lot better shape than when this bug was filed,
but still, odds are that "make test" will fail on the average machine.

I've been collecting a blacklist of bad and/or flaky tests.
Here's one way to use it to measure how unreliable the flaky tests are
on a particular system:

wget -c http://winezeug.googlecode.com/svn/trunk/buildbot/dotests.sh
wget -c http://winezeug.googlecode.com/svn/trunk/buildbot/dotests_blacklist.txt
rm -rf ~/.wine
winetricks nocrashdialog
while true
do
    make testclean
    sh dotests.sh flakytests
done 

Out of 40 or so runs on one system, 
five dll/foo.ok files exhibited crashes or failures:

7 dlls/wininet/tests/urlcache.ok   (bug 28038, workaround: clear cache between
runs)
4 dlls/msctf/tests/inputprocessor.ok (bug 28288)
4 dlls/kernel32/tests/pipe.ok  (bug 28257)
3 dlls/user32/tests/static.ok (bug 20149)
2 dlls/urlmon/tests/url.ok (bug 28108)

So about half of 40 runs of "make test" would probably fail because of 
flaky tests on this machine without manual workarounds.

Your mileage may vary.  Other systems tickle other bugs.

That blacklist and script can also look for tests that have failed on any
of my systems repeatably.
Running 'sh dotests.sh badtests' twice on that same (64 bit) system, I got

      2 dlls/ieframe/tests/ie.ok FAILED; blacklist says BAD64 , see bug 26768 .
      2 dlls/kernel32/tests/process.ok FAILED; blacklist says NOTTY , see bug
28220 .
      2 dlls/oleaut32/tests/tmarshal.ok FAILED; blacklist says BAD64 , see bug
26768 .
      1 dlls/user32/tests/msg.ok FAILED; blacklist says SYS , see bug 12053 .
      2 programs/wscript/tests/run.ok FAILED; blacklist says BAD64 , see bug
28285 .

so "make test" will always fail on this system because of msg tests
(probable workaround: run a blessed window manager) and because
win64 isn't finished.  And a kernel test will always fail because
I tend to redirect the output of "make test".

-- 
Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email
Do not reply to this email, post in Bugzilla using the
above URL to reply.
------- You are receiving this mail because: -------
You are watching all bug changes.



More information about the wine-bugs mailing list