SOC 2009: Application Test Suite

Wed Mar 25 11:47:18 CDT 2009

On Wed, Mar 25, 2009 at 9:47 AM, Francois Gouget <fgouget at free.fr> wrote:
> On Tue, 24 Mar 2009, Austin English wrote:
> [...]
>> I'd like to implement an application test suite. It's something that's
>> been discussed for Wine for quite a while, but has never been put into
>> place.
>
> As mentioned before, check out cxtest. In particular the way they are
> structured, with one script to install the application, and others
> testing particular aspects of it. You will also need to support
> dependencies at some point.

Yes, I've looked at cxtest a bit. I like Jeremy's line of
thinking...keep cxtest for its infrastructure, but move toward AHK,
which gains us windows compatibility.

> But also check out Lei Zhang's experiments with AutoHotKey:
>
> http://code.google.com/p/yawt/

I've looked a bit at this code, should be useful.

> [...]
>> My plan to implement it is to make a script, that can be used
>> standalone, in conjunction with Patchwatcher, or any other script, to
>> test wine against several real world applications. I make a very quick
>> proof of concept (attached), which is based off of winetricks.
>
> One very important feature is that:
>
>
>     A good testing framework *must* work on Windows.

Exactly why Cxtest is not sufficient.

> Why? For the same reason that the conformance tests would not be even
> half as useful as they are if they did not run on Windows:
>
>  * Applications should install the same way in Wine and on Windows. Sure
>   there's bound to be some differences but these should be investigated
>   and either fixed, or documented and worked-around in the test script.

Of course. My very brief initial testing already found a couple of
these sorts of bugs.

>  * This would make it possible to write tests on Windows for things that
>   don't work yet in Wine. This can be particularly powerful in
>   conjunction with Bugzilla: just like we sometimes ask for a
>   conformance test, we could ask for an application test for a
>   given Bugzilla report. Then, sprinkle a few todo_wine and we will
>   immediately know when a bug is fixed.

You read my mind :-)...exactly what I'd like to see. AHK is powerful
in this regard as A) it already works well on windows, B) allows for
gui recording of scripts, which can be adjusted manually and
rewritten. This allows an average user to download AHK (no need to
install, has a portable version you can run), fire up foo.exe which
doesn't work on wine, record a simple test, and file a bug. It greatly
simplifies bug triagers' work, especially for applications in foreign
languages (for some reason, there are quite a few Polish and Slovak
bugs in bugzilla...).

>  * This makes it possible for non-Unix users to contribute. Ideally,
>   once we have developped a test for an open-source Windows
>   application, the developers for that application should be able to
>   use our tests and take over their maintenance.

Of course.

> So AutoHotKey is a step towards that goal. But you will need some
> infrastructure around it to download installers, setting up CDs, maybe
> start and stop virtual machines, and it is this part that worries me. In
> particular I don't think that bash+wget&co is the most appropriate
> toolset. What to use on Windows is a tricky issue though.

Right. For downloading, wget for windows exists, as well as several
other small utilities (ftp is builtin). CDs are one that I've been
thinking about, haven't decided on that yet. Virtual machines would be
useful, unfortunately I don't know much about scripting for those
(shouldn't be too difficult to learn. Hopefully a potential mentor
would have some insight <g>).

For that matter, keep in mind that cxtest is a bash script as well
http://cxtest.ifne.eu:82/transfer/cxtest.sh.

> Other important features:
>  * You should not need to update the scripts for every little change in
>   Wine.

Of course. That is something that I've kept in mind, but not an easy
task. It will take careful planning.

>> 2. Microsoft Office - We've broken this a few times in recent months,
>> so adding a test in daily git for it would help catch regressions
>> sooner.
>
> Just as a note, CodeWeavers tests Microsoft Office every day with
> cxtest. However this is with the CrossOver codebase so we only detect
> WineHQ regressions when it gets merged after each release (so roughly
> every two weeks). In any case it would be nice if these were detected
> sooner and more 'upstream'<g>.

Right. The problem is those tests don't come before wine releases, nor
daily, which allows Office to break in binary releases, not just git.
Knowing the problem exists is only part of the fix, and if that takes
2 weeks or more to find, the installer could break for 2-3 WineHQ
releases.

>> 3. World of Warcraft - I don't play, but A) It's really popular, and
>> B) also gives us a chance to test wininet for regressions.
>> 4. iTunes - this installers been broken a couple times. The bonjour
>> service should allow for testing services related regressions.
>
> The problem with these two is that they change every other day, so that
> you're likely to end up having to update your script pretty often. Also,
> as far as I understand it, Wow is really huge and regularly insists on
> downloading big update patches (which of course you won't have been able
> to checksum). This is not to say it shouldn't be done though.

That's why I put them low on the list ;-). WoW would be nice to have,
but not necessary. iTunes would be very useful, and the problem of
updating daily can be avoided by using mirrors. Verify original
install from Apple.com, get SHA1SUM, then get that version from
http://www.filehippo.com/, verifying filesize and sha1sum.

> Some pointers to past discussions on the subject (far from exhaustive):
> http://www.winehq.org/pipermail/wine-devel/2007-July/057890.html
> http://www.winehq.org/pipermail/wine-devel/2007-March/055174.html

Thanks. I've followed these discussions in the past, but a refresher
is always good.

My perception is that this sort of project is needed pretty badly, but
suffers from 2 main problems: No one wants to do it and those who have
tried in the past aimed to high, crashed, and burned. I think I could
do better because:

I want to do it.

I've tested _tons_ of windows applications (all that bugzilla testing
as good for something).

I've got a more pragmatic view of what's possible. I'm not aiming to
make winetest version 2.0 for windows. The AHK scripts will be
runnable on windows, sure. But the main bulk of the script will be
built for Wine/Unix. While that may not be ideal (the ahk scripts may
not be tested daily on windows), it shouldn't be too hard for someone
who wants to implement that on windows to do so. Windows doesn't
change all that often, Wine does, so it follows that we should test it
more often. If the test passes on Windows originally, and a curious
party tests it every month or two (or more likely, when a bug is
found/fixed), that should be plenty.

This wouldn't be the panacea of regression testing for Wine, but would
be a major step in that direction.

-- 
-Austin