[Bug 31787] New: Run the tests in Wine
wine-bugs at winehq.org
wine-bugs at winehq.org
Mon Sep 24 18:46:48 CDT 2012
http://bugs.winehq.org/show_bug.cgi?id=31787
Bug #: 31787
Summary: Run the tests in Wine
Product: Wine-Testbot
Version: unspecified
Platform: x86
OS/Version: Linux
Status: NEW
Severity: normal
Priority: P2
Component: unknown
AssignedTo: wine-bugs at winehq.org
ReportedBy: fgouget at codeweavers.com
Classification: Unclassified
The Wine TestBot runs the conformance tests on Windows VMs before they are
committed to Wine. This greatly helped improve the quality of our conformance
tests.
The next step is making sure that the conformance tests work reliably in Wine
too, and that Wine patches don't break them. The hope is that this will help us
improve the state of the WineTest results:
http://test.winehq.org/data/
Doing so will be much more processor intensive than the current process:
* Currently only the conformance test patches are tested whereas we would need
to test almost every single patch (we could ignore some documentation patches
for instance).
* Currently we only have to generate a binary for the one conformance test
that's impacted. That's quick. Testing all Wine patches will require
recompiling Wine every time which takes more time, even with ccache.
* A Wine patch can have wide-ranging effects. For instance a patch to ntdll
could impact pretty much any conformance test. That means rerunning all of them
for every patch... unless we find reliable ways of pruning them (patches to
conformance tests would be an obvious optimization).
* Like for Windows we will need to test various configurations: different
locales; 32, 64 and 32+64 bit Wine builds; different sound backends; possibly
different Linux distributions; if possible FreeBSD and Solaris too (I'm leaving
Mac OS X for another bug). In some cases the same binaries can be used for
multiple tests (e.g. if only the locale changes) but in other cases not.
* Also for the current Wine TestBot an unreliable test only impacts people
modifying that test. But the above point means an unreliable test would impact
most Wine patches, leading to lots of patches being rejected. So this means
unreliable tests become a big issue and need to be handled in some way: fixed
or blacklisted or still run but not considered cause for rejection (so one can
notice when they get fixed).
Then there is the question of whether this should be done by modifying the Wine
TestBot or by using the BuildBot framework:
http://trac.buildbot.net/wiki/AboutBuildbot
Dan Kegel produced a proof of concept system based on BuildBot:
http://wiki.winehq.org/BuildBot
An issue is that without significant integration work, having both a
WineTestBot and BuildBot system would be annoying:
* The Wine TestBot currently has its own user database and BuildBot would add
another one. So it would be one more login to manage for developers unless we
manage to integrate either with another user database (e.g. through LDAP).
* Developers would have to submit patches to both sites and check the results
on both.
* Integration with the Wine Patches site (http://source.winehq.org/patches/)
would need to work with both systems.
--
Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email
Do not reply to this email, post in Bugzilla using the
above URL to reply.
------- You are receiving this mail because: -------
You are watching all bug changes.
More information about the wine-bugs
mailing list