We *really* need a development model change !

[Andriy Palamarchuk]

--- Alexandre Julliard <julliard at winehq.com> wrote:
> Andriy Palamarchuk <apa3a at yahoo.com> writes:
> 
> > Always succeed *under Windows*. Do you really,
> really,
> > really think all the tests will succeed under Wine
> > from day 1 and we will be able to maintain them
> > failure-free?
> 
> Absolutely. There's a very simple way of enforcing
> that: I'm not
> comitting anything that causes make test to fail.
>
> > The value of unit tests is exactly in failures!
> The
> > more failures of unit tests we have - the better
> test
> > developers do their work.
> 
> Well, I guess it's a philosophical point, but for me
> the value of the
> tests is for regression testing. If you allow the
> tests to fail you'll
> pretty soon have 90% of the tests fail somewhere,
> and this is
> completely useless except maybe as a list of things
> we still have to
> do. While if the tests usually succeed, as soon as
> something fails you
> know there's a regression and you have to fix it.

As Francois mention, this is why TODO tests exists.
Even without TODO tests it is not wise to reject a
perfectly correct patch from a Windows developer who
even does not have Wine.

Without TODO construction compromise is still
possible. E.g we can use conditional compilation or
execution to select only tests which succeed for
finding regressions. I believe there will be somebody,
interested in mining "dirty" tests.
Other option is to store failture lists and compare
them from time to time in a search of regressions.

> What you can do with my make test patch is run make
> test -k first, let
> it go through everything, and then run make test
> again and it will
> only run the tests that failed the first time, or
> that have been
> modified. This is a major gain. It could certainly
> be done some other
> way too without using make, but AFAICS your test
> harness would force
> to run all tests all the time (or to disable them
> manually). This is
> not acceptable once you have a large number of
> tests.

I don't see poing in selecting subsets of tests. From
experience - even with pretty big number of tests it
does not take long time to execute them.

> OTOH a Wine test suite can happen (I hope) because
> this is something
> Wine developers need when they write code, so there
> is at least some
> motivation for them to write tests.

Exactly! I do not want to spend resources of Wine
project on some "nice documentation". On the contrary,
goal of this change is to invite external resources.

Andriy Palamarchuk

__________________________________________________
Do You Yahoo!?
Send your FREE holiday greetings online!
http://greetings.yahoo.com