We *really* need a development model change !

[Andreas Mohr]

On Wed, Dec 26, 2001 at 10:07:20AM -0800, Andriy Palamarchuk wrote:
> Andreas Mohr wrote:
> > I guess we really should change our development
> model from trying tons of
> > programs to *systematically* testing functions and
> Windows mechanisms now.
> > If we can show everyone where stuff is failing, it
> might be a lot easier
> > to attract new people.
> 
> I *completely* support this idea. Benefits of such
> test suite are enormous. Existing developers can
> contribute a lot by adding test snippets for the
> functions they create. Now they create such snippets
> anyway and throw them away.
Ah, good ! :-)
Exactly. A lot of people create test code e.g. for undocumented functions etc.
By adding a *slight* bit more work, they'd have a test for this function.

> Comments:
> - Don't want to reinvent the weel. Is there any
> existing test suite framework we can use? Sorry, I
> can't suggest any for C but I'm very impressed with
> JUnit in Java. It is even Ok if the framework is GPLed
> or LGPLed - I don't think any company will make
> buziness based on the test suite.
Hmm, good question. I don't know of any, but we should probably do some more
research. After all it's about 12000 functions, so we should get it right.

> - I /personally/ prefer CL interface only for such
> suite
Yes, yes, yes. *Much* easier to use. That's why I did exactly that kind of
thing.

> - it would be better if the suite print summary
> information and information about failed tests only
Yep. Current output is something like:
WINETEST:test:Loader_16:LoadModule:FAILED:01:[retval]
WINETEST:test:Loader_16:LoadModule:FAILED:12:[retval]
WINETEST:test:Loader_16:LoadModule:FAILED:13:[retval]

or, in case of success, only:
WINETEST:test:Loader_16:LoadModule:OK

(yeah, I know, wishful thinking ;-)

This output is pretty useful, I think:
It can be parsed *very* easily, and grepping for regressions is also pretty
easy.

"WINETEST" exists to be able to distinguish this output from bogus Wine
messages,
"test" indicates that this is a test line output versus a warning message or
similar output,
"Loader_16" indicates testing of 16bit loader functionality,
"LoadModule" - well... ;-)
"FAILED" - obvious
"01" - test number 01 failed.
"[retval]" - contains the (wrong) return value of the function, if applicable.

BTW, I think having a test suite wouldn't be about hunting regressions
at first: just look at my LoadModule16 example and you'll see that we're
still quite far from hunting regressions *only*.
My guess is that we'll be shocked at how many functions fail in how many ways.

> - make the test suite more "visible" for existing
> developers. Ask them to run the test suite before
> submitting a patch?
No, I don't think so.
I think it suffices if Alexandre runs the test suite before or after every
large commit cycle.
That way he'd be able to back out problematic patches.
Asking developers to run the *whole* test suite for each patch could be
pretty painful.

> - I think the suite test will consist from a few
> separate applications because different tests may have
> different requirements to GUI configuration,
> processes, etc. We need a way to run all the
> applications in one batch.
Exactly. Which is why I really prefer simple text output. IMHO it's the only way
to go.

> - define variable which indicates whether the suite
> runs under Wine. Such indicator can be used for Wine
> "white-box" testing.
Hmm, yes, that might be useful.
We'd also need to pass a winver value to the test suite via command line
in order to let the test app adapt to different windows environments
(and thus also to different wine --winver settings !).

> - it would be greate to have functionality to support
> output comparison? For some functionality it is easier
> to write tests to compare output instead of doing
> explicit checks (e.g. tests, involving a few
> processes). The output can be redirected to file and
> files compared. If we use files we need to store files
> for Wine and a few versions of Windows :-(
Hmm, I don't quite get what exactly you're talking about.

> - the suite applications size will be pretty big. Is
> it better to move it to separate CVS tree?
Yep, I'd say so. There definitely is no business for it to reside in the main
Wine tree.

> - what about running the suite weekly (or daily)
> automatically and publishing the results to
> wine-devel?
Good idea ! Might prove worthwhile.

> - most developers on this list have access to one
> version of Windows. Is it difficult to create "testing
> farm" with remote access to a few versions of windows?
> This would help developers to test their code on a few
> platforms. Existing environments in the companies,
> involved in the project can be used.
Hmm, why ?
The idea is that hundreds (or hopefully thousands ?) of volunteer Windows
developers create bazillions of test functions for specific API functions.
That will happen on specific Windows version only, of course.
Now we have a test framework for a specific API function on a specific
Windows version.
Now if there are behavioral conflicts on different Windows versions
(functions behave differently), then I guess people will notice immediately
and fix the test function immediately to support different behaviour of
different windows versions.
--> no problem at all.

> - I remember long time ago there was a post on
> wine-devel about using Perl or Perl-like language for
> unit testing.
> What is current status of that project?
Hmm. That'd be programs/winetest/, right ?

-- 
Andreas Mohr                        Stauferstr. 6, D-71272 Renningen, Germany
Tel. +49 7159 800604                http://home.nexgo.de/andi.mohr/