SOC 2009: Application Test Suite

[Roderick Colenbrander]

Hi Austin,

Not sure if you are aware of it but there is also cxtest which was written
by codeweavers under the gpl. See http://cxtest.ifne.eu:82/ it seems they
(still?) use it regulary to track regressions. I haven't looked at it and
don't know that autohotkey stuff but how do both differ? Wouldn't it be
better to continue with cxtest? Or perhaps it would be better to use
autohotkey (assuming it is a widely used app) and extend it so that cxtest
can be dropped.

Roderick

On Wed, Mar 25, 2009 at 12:33 AM, Austin English <austinenglish at gmail.com>wrote:

> Howdy,
>
> I'm planning on applying for Google Summer of Code 2009. I'm pretty
> sure most people reading this list already know me, but for those that
> don't, I frequently triage bugs in Bugzilla, help users on the forum,
> and a few other things. I've also done quite a bit of work testing
> wine on more obscure OS's (OpenSolaris, FreeBSD, OS X,  NetBSD,
> OpenBSD).
>
> I'd like to implement an application test suite. It's something that's
> been discussed for Wine for quite a while, but has never been put into
> place. I took a bit of time a while back and hacked couple a couple
> quick scripts as a test, and got a script to automatically install
> Firefox 3 in Wine using AutoHotKey (http://www.autohotkey.com/). I've
> made a couple more scripts as well, but they don't match Windows
> behavior because of bug (http://bugs.winehq.org/show_bug.cgi?id=10147)
> hint, hint (disclaimer, I haven't tested that script in a while, been
> busy with exams, so that bug might be fixed...I don't think it has
> though).
>
> My plan to implement it is to make a script, that can be used
> standalone, in conjunction with Patchwatcher, or any other script, to
> test wine against several real world applications. I make a very quick
> proof of concept (attached), which is based off of winetricks. It's
> _REALLY_ rough at the moment, (I haven't had time to mess with it
> lately), but demonstrates the idea. The script setups a clean
> WINEPREFIX, installs autohotkey, removes the Z: symlink for braindead
> installers that search the entire hard drive, and sets up a link to
> it's install cache (essentially winetrickscache). From there, it makes
> a copy of this prefix, so on subsequent installers/applications, we
> can restore the clean prefix and save a few seconds.
>
> From there, so far, it only installs firefox3. The plan is to add
> several dozen different tests, for various functions of wine.
> Obviously, I'd like to start with platinum rated applications, to
> prevent regressions. From there, the current list of
> applications/functions to target are:
> 1. Firefox3 - useful for testing lots of other things, e.g.,
> java/flash, etc. Easy to test.
> 2. Microsoft Office - We've broken this a few times in recent months,
> so adding a test in daily git for it would help catch regressions
> sooner.
> 3. World of Warcraft - I don't play, but A) It's really popular, and
> B) also gives us a chance to test wininet for regressions.
> 4. iTunes - this installers been broken a couple times. The bonjour
> service should allow for testing services related regressions.
> 5. Adobe suite - there haven't been too many regressions here that I
> can remember, but it's definitely an area we should watch out for.
>
> Of course, we don't want to only test the installer's keystrokes/mouse
> inputs, so SHA1 checksumming of installed files would verify against
> corrupted files/setupapi regressions/etc. Once a few installer tests
> are in and working well, I plan to extend the tests further to test
> running the application, e.g, for photoshop by opening a file,
> inserting a circle, and saving it. This is a bit harder to pull off in
> Autohotkey in a way that's portable, so I'm saving that sort of thing
> for later.
>
> Also interesting would be a real application's test suite. Dan
> suggested using Google Chrome's, since it's squeaky clean after using
> lots of Valgrind and Purify. That would be good, but I'd put that
> after getting the test suite to pass in Wine.
>
> This should be very doable using Autohotkey's scripting functionality
> (pretty extensive - http://www.autohotkey.com/docs/Functions.htm,
> http://www.autohotkey.com/docs/Variables.htm) and good old fashioned
> shell scripting. I'm very familiar with bourne scripting, and
> currently maintain winetricks while Dan's taking a Wine vacation. I'm
> currently working on a build script to build Wine on several OS's, as
> well as running and submitting the conformance tests
> (http://winezeug.googlecode.com/svn/trunk/build_script/daily.sh). [I
> got tired of maintaining the script over several different
> OS's/computers]. The upside is it would be easy to add on appinstall
> to this daily script. I typically set it running before leaving home
> for work/school. It spends an hour or so building wine and running the
> tests in a few different configurations. Adding appinstall would allow
> my computer to daily test several applications for installation,
> running, etc. This of course isn't as useful as Patchwatcher's testing
> of each single patch, but I don't currently have a spare machine that
> it can test on. Getting a script and tests ready to do so, however,
> would be great for Wine. Besides, I can easily setup the script to
> upload the results to a webserver, so we can daily see that each test
> is passing. If a test is failing, the information would then be
> publicly available for anyone to run 'git bisect'.
>
> Wow, this turned into a really long e-mail, and I've got an exam to
> study for. I'd appreciate any comments, and let me know if there are
> any questions.
>
> --
> -Austin
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20090325/2522ee5f/attachment.htm>