We *really* need a development model change !

[Andriy Palamarchuk]

--- Francois Gouget <fgouget at free.fr> wrote:
[...]

> A test unit can also exercise more than one aspect
of 
> one or more APIs. But, as a rule of thumb, a
specific 
> test should not exercise more than a couple to a 
> handful related APIs (or up to a dozen in extreme
> cases). Also, more than one test could exercise 
> different aspects a given API.

I don't see a problem in having complex, task-oriented
tests. Simple tests check one module, coplex tasks
help to check integration between components. Both are
needed. I especially like to create tests which
implement long data conversion chains where output
should be equal to input :-)
I still want to have well-structured tests.

> Wine tests can write their output in any form they 
> like. The only important criteria are:

One more criteria - restore the system state (if
possible). We don't want to have misconfigured machine
after running the test suite.

> To each test we associate a file containing the 
> reference output for
> that test.

Maintaining reference output file(s) is difficult
because of:
- EOLs conversions
- keep all the files in sync
- output files can be huge
- test suite can have different output for different
versions of Windows
- one person can't run the suite on all Windows
platforms, those who can, are afraid to touch fragile
system of the output reference files.

I suggest to use explicit checks and print descriptive
messages in case of falure. I agree, this approach is
more labour-intensive, especially for tests using IPC.
It is also much more maintainable as soon as you coded
it. Everything, including differences between
different Windows platforms is documented directly in
the code! This gives much better control. E.g, it is
possible to comment-out part of the test, still
getting meaningful results what worked, what did not.

One more idea about reference output - it can be
included in Perl script as in-line constant, so we can
keep the output in the same file as the test itself.

> Each test should contain a section that looks 
> something like:
>
> # @START_TEST_COVERAGE@
> # kernel32.CreateProcess
> # kernel32.GetCommandLineA
> # kernel32.GetCommandLineW
> # __getmainargs
> # __p__argc
> # __p_argv
> # __p_wargv
> # @END_TEST_COVERAGE@

I prefer to have descriptive comments and failure
messages. It will be difficult to keep the description
you suggest in sync with the test implementation.

> * TEST_WINVER
>    This contains the value of the '-winver' Wine 
> argument, or the
> Windows version if the test is being run in Windows.

> We should mandate the use of specific values of 
> winver so that test don't have to
> recognize all the synonyms of win2000 (nt2k...),
etc.
>    (Do we need to distinguish between Windows and 
> Wine?)

Yes, we do. Sometimes we have implemented behavior
only for one Windows version, e.g. NT. In this case
switch -winver won't affect Wine behaviour.

> * TEST_BATCH
>    If true (equal to 1) or unset, then the test 
> should assume that it is being run from within the 
> test framework and thus that it should be
> non-interactive. If TEST_BATCH is set to 0, then the

> test can assume that it is being run in interactive 
> mode, and thus ask questions to the
> user. Of course most tests will simply behave 
> identically in both cases,

I strongly recommend to use only batch version. If
somebody has reasons to play with different forms of
input, he can use any of following easy options:
a) hardcode it - original test is improved ;-)
b) take the test code to his interactive application
c) add interaction with user to his local copy of the
tests

> Running tests
> -------------
> In Wine:
> 
>    'make tests' seems the best way to do things.
>    But some tests may need to create windows. For 
> instance I have a DIB test that creates a window, 
> draws a number of DIBs in it and checks the
> bitmap bits of these DIBs and then exits. Thus it is

> a non-interactive test. I am not really sure whether

> the window actually needs to be made
> visible or not, but even if this particular exemple 
> does not require it,
> I suspect that othersm checking for message
sequences 
> for instance, may need to make the window visible.

I have tests which show window to use window
messaging.  Idea about tests separation to cui and gui
looks good.
You can run cui tests while doing something else.

> In Windows:
>
>  Hmmm, not sure how that is done. Run
'winetest.exe'?

Something like:
perl run_all_tests.pl

The unit test application we develop will be a test of
Win32 API in general, useful for all implementations.
I'm for keeping this test in separate directory tree, 
not mixing it with Wine files.

Andriy Palamarchuk

__________________________________________________
Do You Yahoo!?
Send your FREE holiday greetings online!
http://greetings.yahoo.com