SOC 2009: Application Test Suite

[Francois Gouget]

On Tue, 24 Mar 2009, Austin English wrote:
[...]
> I'd like to implement an application test suite. It's something that's
> been discussed for Wine for quite a while, but has never been put into
> place.

As mentioned before, check out cxtest. In particular the way they are 
structured, with one script to install the application, and others 
testing particular aspects of it. You will also need to support 
dependencies at some point.

But also check out Lei Zhang's experiments with AutoHotKey:

http://code.google.com/p/yawt/


[...]
> My plan to implement it is to make a script, that can be used
> standalone, in conjunction with Patchwatcher, or any other script, to
> test wine against several real world applications. I make a very quick
> proof of concept (attached), which is based off of winetricks.

One very important feature is that:


     A good testing framework *must* work on Windows.


Why? For the same reason that the conformance tests would not be even 
half as useful as they are if they did not run on Windows:

 * Applications should install the same way in Wine and on Windows. Sure 
   there's bound to be some differences but these should be investigated 
   and either fixed, or documented and worked-around in the test script.

 * This would make it possible to write tests on Windows for things that 
   don't work yet in Wine. This can be particularly powerful in 
   conjunction with Bugzilla: just like we sometimes ask for a 
   conformance test, we could ask for an application test for a 
   given Bugzilla report. Then, sprinkle a few todo_wine and we will 
   immediately know when a bug is fixed.

 * This makes it possible for non-Unix users to contribute. Ideally, 
   once we have developped a test for an open-source Windows 
   application, the developers for that application should be able to 
   use our tests and take over their maintenance.

So AutoHotKey is a step towards that goal. But you will need some 
infrastructure around it to download installers, setting up CDs, maybe 
start and stop virtual machines, and it is this part that worries me. In 
particular I don't think that bash+wget&co is the most appropriate 
toolset. What to use on Windows is a tricky issue though.

Other important features:
 * You should not need to update the scripts for every little change in 
   Wine.


> 2. Microsoft Office - We've broken this a few times in recent months,
> so adding a test in daily git for it would help catch regressions
> sooner.

Just as a note, CodeWeavers tests Microsoft Office every day with 
cxtest. However this is with the CrossOver codebase so we only detect 
WineHQ regressions when it gets merged after each release (so roughly 
every two weeks). In any case it would be nice if these were detected 
sooner and more 'upstream'<g>.


> 3. World of Warcraft - I don't play, but A) It's really popular, and
> B) also gives us a chance to test wininet for regressions.
> 4. iTunes - this installers been broken a couple times. The bonjour
> service should allow for testing services related regressions.

The problem with these two is that they change every other day, so that 
you're likely to end up having to update your script pretty often. Also, 
as far as I understand it, Wow is really huge and regularly insists on 
downloading big update patches (which of course you won't have been able 
to checksum). This is not to say it shouldn't be done though.


Some pointers to past discussions on the subject (far from exhaustive):
http://www.winehq.org/pipermail/wine-devel/2007-July/057890.html
http://www.winehq.org/pipermail/wine-devel/2007-March/055174.html


-- 
Francois Gouget <fgouget at free.fr>              http://fgouget.free.fr/
  Nouvelle version : les anciens bogues ont été remplacés par de nouveaux.