[Bug 31787] New: Run the tests in Wine

[wine-bugs at winehq.org]

http://bugs.winehq.org/show_bug.cgi?id=31787

             Bug #: 31787
           Summary: Run the tests in Wine
           Product: Wine-Testbot
           Version: unspecified
          Platform: x86
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: unknown
        AssignedTo: wine-bugs at winehq.org
        ReportedBy: fgouget at codeweavers.com
    Classification: Unclassified


The Wine TestBot runs the conformance tests on Windows VMs before they are
committed to Wine. This greatly helped improve the quality of our conformance
tests.

The next step is making sure that the conformance tests work reliably in Wine
too, and that Wine patches don't break them. The hope is that this will help us
improve the state of the WineTest results:
   http://test.winehq.org/data/

Doing so will be much more processor intensive than the current process:
* Currently only the conformance test patches are tested whereas we would need
to test almost every single patch (we could ignore some documentation patches
for instance).

* Currently we only have to generate a binary for the one conformance test
that's impacted. That's quick. Testing all Wine patches will require
recompiling Wine every time which takes more time, even with ccache.

* A Wine patch can have wide-ranging effects. For instance a patch to ntdll
could impact pretty much any conformance test. That means rerunning all of them
for every patch... unless we find reliable ways of pruning them (patches to
conformance tests would be an obvious optimization).

* Like for Windows we will need to test various configurations: different
locales; 32, 64 and 32+64 bit Wine builds; different sound backends; possibly
different Linux distributions; if possible FreeBSD and Solaris too (I'm leaving
Mac OS X for another bug). In some cases the same binaries can be used for
multiple tests (e.g. if only the locale changes) but in other cases not.

* Also for the current Wine TestBot an unreliable test only impacts people
modifying that test. But the above point means an unreliable test would impact
most Wine patches, leading to lots of patches being rejected. So this means
unreliable tests become a big issue and need to be handled in some way: fixed
or blacklisted or still run but not considered cause for rejection (so one can
notice when they get fixed).

Then there is the question of whether this should be done by modifying the Wine
TestBot or by using the BuildBot framework:
   http://trac.buildbot.net/wiki/AboutBuildbot

Dan Kegel produced a proof of concept system based on BuildBot:
   http://wiki.winehq.org/BuildBot

An issue is that without significant integration work, having both a
WineTestBot and BuildBot system would be annoying:
* The Wine TestBot currently has its own user database and BuildBot would add
another one. So it would be one more login to manage for developers unless we
manage to integrate either with another user database (e.g. through LDAP).
* Developers would have to submit patches to both sites and check the results
on both.
* Integration with the Wine Patches site (http://source.winehq.org/patches/)
would need to work with both systems.

-- 
Configure bugmail: http://bugs.winehq.org/userprefs.cgi?tab=email
Do not reply to this email, post in Bugzilla using the
above URL to reply.
------- You are receiving this mail because: -------
You are watching all bug changes.