SOC 2009: Application Test Suite
fgouget at free.fr
Wed Mar 25 09:47:51 CDT 2009
On Tue, 24 Mar 2009, Austin English wrote:
> I'd like to implement an application test suite. It's something that's
> been discussed for Wine for quite a while, but has never been put into
As mentioned before, check out cxtest. In particular the way they are
structured, with one script to install the application, and others
testing particular aspects of it. You will also need to support
dependencies at some point.
But also check out Lei Zhang's experiments with AutoHotKey:
> My plan to implement it is to make a script, that can be used
> standalone, in conjunction with Patchwatcher, or any other script, to
> test wine against several real world applications. I make a very quick
> proof of concept (attached), which is based off of winetricks.
One very important feature is that:
A good testing framework *must* work on Windows.
Why? For the same reason that the conformance tests would not be even
half as useful as they are if they did not run on Windows:
* Applications should install the same way in Wine and on Windows. Sure
there's bound to be some differences but these should be investigated
and either fixed, or documented and worked-around in the test script.
* This would make it possible to write tests on Windows for things that
don't work yet in Wine. This can be particularly powerful in
conjunction with Bugzilla: just like we sometimes ask for a
conformance test, we could ask for an application test for a
given Bugzilla report. Then, sprinkle a few todo_wine and we will
immediately know when a bug is fixed.
* This makes it possible for non-Unix users to contribute. Ideally,
once we have developped a test for an open-source Windows
application, the developers for that application should be able to
use our tests and take over their maintenance.
So AutoHotKey is a step towards that goal. But you will need some
infrastructure around it to download installers, setting up CDs, maybe
start and stop virtual machines, and it is this part that worries me. In
particular I don't think that bash+wget&co is the most appropriate
toolset. What to use on Windows is a tricky issue though.
Other important features:
* You should not need to update the scripts for every little change in
> 2. Microsoft Office - We've broken this a few times in recent months,
> so adding a test in daily git for it would help catch regressions
Just as a note, CodeWeavers tests Microsoft Office every day with
cxtest. However this is with the CrossOver codebase so we only detect
WineHQ regressions when it gets merged after each release (so roughly
every two weeks). In any case it would be nice if these were detected
sooner and more 'upstream'<g>.
> 3. World of Warcraft - I don't play, but A) It's really popular, and
> B) also gives us a chance to test wininet for regressions.
> 4. iTunes - this installers been broken a couple times. The bonjour
> service should allow for testing services related regressions.
The problem with these two is that they change every other day, so that
you're likely to end up having to update your script pretty often. Also,
as far as I understand it, Wow is really huge and regularly insists on
downloading big update patches (which of course you won't have been able
to checksum). This is not to say it shouldn't be done though.
Some pointers to past discussions on the subject (far from exhaustive):
Francois Gouget <fgouget at free.fr> http://fgouget.free.fr/
Nouvelle version : les anciens bogues ont été remplacés par de nouveaux.
More information about the wine-devel