We *really* need a development model change !

[Ulrich Weigand]

Alexandre Julliard wrote:
> Dan Kegel <dank at kegel.com> writes:
> > That's great to hear, but I think you have to modify your statement a
> > bit -- you may want to commit new tests don't yet pass,
> > if they show a real flaw in Wine.
> 
> In that case the test should use a TODO mechanism or equivalent, and
> it must still be possible to run make test without failure (but there
> would be an option to switch the failures on if you want).

The dejagnu test harness that is used by e.g. the gcc test suite
allows to classify a test case as 'expected to fail'.

When you run the test, every test case either passes or fails,
which results in a classification into four subsets:

  PASS   Test case was expected to pass, and it did
  FAIL   Test case was expected to pass, but failed
  XPASS  Test case was expected to fail, but passed
  XFAIL  Test case was expected to fail, and it did

Only a test case in the FAIL category causes the whole test run
to fail, and is reported even in the test run summary.  The other
categories are only reported as total numbers.

If you are getting nonzero FAIL numbers, you have introduced a
regression.  Nonzero PASS and XFAIL numbers are expected; if you
get nonzero XPASS numbers you might look at the cases in question
and decide whether you want to remove the 'expected to fail' flag.

This system works quite well in my experience with gcc, maybe something
like this could be implemented for Wine as well ...

Bye,
Ulrich

-- 
  Dr. Ulrich Weigand
  weigand at informatik.uni-erlangen.de