We *really* need a development model change !

[Alexandre Julliard]

Dan Kegel <dank at kegel.com> writes:

> That's great to hear, but I think you have to modify your statement a
> bit -- you may want to commit new tests don't yet pass,
> if they show a real flaw in Wine.

In that case the test should use a TODO mechanism or equivalent, and
it must still be possible to run make test without failure (but there
would be an option to switch the failures on if you want).

> That means you probably want to live with less than 100%  success rates.
> The important thing when committing a new change to Wine
> (as opposed to a change to the test suite) is that it not
> cause any *new* failures.  I bet that's what you meant.

No, what I mean is that you can't spot new failures if every test run
shows hundreds of existing ones. The only way to find new failures is
if you can do a successful test run before a change.

Imagine that you have 1000 tests, and a typical run shows 250
failures. Then you make a change, and you now see 248 failures. Does
it mean you fixed 2 bugs, or does it mean you fixed 5 and introduced 3
new ones?  You have no way of knowing without going through the 250
failures one by one. This is clearly not possible.

-- 
Alexandre Julliard
julliard at winehq.com