Wine regression testing: PIT_

[Molle Bestefich]

Regression testing with Wine is a pain in the butthole.

For very bad reasons: Stuff that is simple to fix, but haven't been.

I'd like to help improve that situation.


Example:
========

I want to find which patch ruins an application.
The application, according to one note, worked in 2003.
I decide to try and see how it really runs with the 2003 version of
Wine mentioned in the note.

I then spend several days pulling various releases, just trying to compile them.
They all fail miserably, because some fix required to compile
correctly with newer Linux versions is /missing/.

For example,
 - any Wine before 2004-01-02 won't work because it won't compile
against newer ALSA versions.
 - any Wine before 2003-12-01 needs patches to the build system before
'make depend' will even run.

I've been at it for a week and I've managed to fix the above two.
_They're really simple things, but they take a lot of time to figure out._


Currently I'm working with two other problems:
 - Wine fails to correctly invoke the debugger. Winedbg shows 'Usage:
blah blah'.  Seems 'wine' calls it with incorrect parameters.
 - Wine fails to start any application, complaining about 'kthreads'
something or the other.

I imagine that if I spend a few more weeks on the above problems, I
can find patches in Wine CVS that fixes the exact problems I'm seeing.

I've already wasted a week of my time on problems that someone else
already fixed.  Spending two or more weeks doing this is not exactly
on my fun-things-to-do-before-I-die list.  I'd like to see a general
solution to this entire problem.



What can be done to alleviate?
==============================

Once I find a patch which makes an earlier version actually compile,
I'd like to propose that for a backport.

If approved/applied, the next person that checks out the release would
check out a version containing the backported patches.

That would fix the problem and allow people to do proper regression
testing, so developers and QA people can spend their time doing
development and QA instead of redoing already done effort trying to
get an older version to run.

I'm not sure how this would work with CVS.  I'm from an SVN world
where tagging is a 0-cost tree-copy operation, and any patches can be
applied to the tag with history and all showing that the tag's been
modified.  If it can't work with CVS, perhaps the tags can be created
in Troy's SVN repo and patches applied there?  Not sure.  Suggestions,
please, people.