We *really* need a development model change !

[Andriy Palamarchuk]

--- Alexandre Julliard <julliard at winehq.com> wrote:
> Andriy Palamarchuk <apa3a at yahoo.com> writes:

[...]
> But adapting the framework to do what we want is IMO
> more work than
> simply reimplementing from scratch the few features
> that we actually
> need. We don't really gain anything by making the
> effort of reusing
> all that code, since we don't need most of it.

Could you, please, list the additional features we
need? I'll try to estimate amount of work necessary to
implement them in Test::Harness.

Do you want to use architecture, completely different
from Test::Harness + Test::Simple modules or you only
want to replace Test::Harness?

Existing architecture:
Individual test scripts (executables) are very simple
(use Test::Simple). They only print messages like "ok
4 - action SPI_FOO # TODO not implemented" for each
check.
Test::Harness module parses output, has logic to
analyze test results, creates overall report and
details of failures, including crashes.

> I don't want to know about the details of
> the 250 individual
> checks.

You got an impression that Test::Harness does not
report individual failures. Sorry, I gave too simple
demo.

Example of Test::Harness output for a few failures:

test2.p.............NOK 2#     Failed test (test2.pl
at line 8)              
#          got: '1'
#     expected: '0'
test2.p.............NOK 3#     Failed test (test2.pl
at line 9)              
#          got: '1'
#     expected: '0'
test2.p.............NOK 4#     Failed test (test2.pl
at line 10)             
test2.p.............ok 8/8# Looks like you failed 3
tests of 8.

[... summary report output is skipped ...]

Does this output look closer to the one you want?
Let me know if you need any other information.

> > 2) mixed C/Perl environment will be used for the
> tests
> > development. Choosing the tool is a matter of
> personal
> > preferences.
> 
> I don't think I agree. For me the value of Perl is
> in that it makes it
> trivial to take the suite over the Windows; but if
> half the tests are
> in C we lose this advantage, and then we might as
> well do everything
> in C.

Sorry, misinterpreted your statement that threads
won't be used for tests in Perl. Could you give your
vision when C is used?

> What I want is a make-like
> system that keeps
> track of which tests have been run, which ones need
> to be re-run
> because they have been modified etc.

This usage of make is fine with me. I just want to
separate unit tests from main code and have
centralized control. You still can call subset of unit
tests from the build process.

> You cannot put the whole test suite in a single
> application, you need
> to split things up some way. A decent test suite
> will probably be
> several times the size of the code it is testing;
> you don't want that
> in a single application.

We were not careful about the terms. Yes, the tests
will be bunch of the executables. I assumed these
executables to be parts of one test application.
Completely agree with you.

> > You preferred to have unit tests
> > spreaded over the Wine directory tree. The main
> > argument for this was possibility of running
> subsets
> > of test.
> 
> No, the argument is modularity.

I'm all for modular unit tests and there are a few
ways to divide the tests between modules.

> Then when you change something in kernel32 you can
> change the test
> that is right next to it, run make test in the
> kernel32 directory and
> have it re-run the relevant tests, and then do cvs
> diff dlls/kernel32
> and get a complete diff including the code and the
> test changes.

Agree, this is not so convenient for separate test
application.

Separate unit tests application has its own
advantages:
1) Separate distributions are used for Wine and the
test applications. The only case when we want them
together - Wine development under *nix. In all other
cases we need only one of them.
2) The unit tests will be mostly developed under
Windows. The unit tests build process has extra
Windows compatibility requirements.
3) Having unit tests as a separate application we
create possibilities to collaborate with other W32
implementation OS projects (ODIN comes to mind).

All the tasks we mentioned can be implemented with any
approach. I believe the tasks you described will be
not much more difficult with my approach, especially
if directory tree has parallel structure.

Andriy Palamarchuk

__________________________________________________
Do You Yahoo!?
Send FREE video emails in Yahoo! Mail!
http://promo.yahoo.com/videomail/