Babystep: Testing framework

[Jeremy White]

> Francois Gouget <fgouget at free.fr> writes:
> 
> >    So the test has to do the diff itself. And no calling 'diff' because
> > it's not portable :-(
> 
> But you shouldn't need diff at all. You just have to write the test
> slightly differently to have checks instead of printfs.

    The normal behaviour of the tests is that they are
oblidged to provide an 0 exit code on success; I think that
is an excellent standard.
    However, having the diff feature allows us to more
rapdily adapt existing programs to become tests.  
Since it's done (and it's trivial code), I don't see
the harm in leaving in the feature.  We can hide it/discourage
in in the (as yet unwritten) doco if you like.

> 
> >    A C test should just be a .spec file and a .c file. If they are all
> > handled the same way it's actually quite simple. In 'Programming Windows
> > 98' I have 148 executables/libraries and 171 C files. It's quite
> > manageable.
> 
> I'm not sure about that; if you run make in libtest/ you end up with
> 70 files and more than 7Mb of disk space, for just 10 nearly empty
> programs. And the makefile is a big mess too.

    The entire samples directory that I submitted consisted 
of 2,705 bytes.  For all CVS checkouts and 'normal' Wine use, 
there will be an insignificant penalty.
    And yes, in the framework I submitted, if you choose to
run 'make test', it chews disk space (and the Winelib tests
are fairly slow to launch).  But the Makefile.in is, IMO,
clean and tight, and easy to replicate.  
And we don't *have* to use Winelib/C tests - the Perl
tests work fine.
    Further, if we had 70 regression tests, that would
be exactly 70 more than we have now!  We can fix the big
bloatedness later.

    Finally, the more I think about my structure of using
the '.test' files, the more I believe it is the right solution,
because I think it gives us great long term flexibility.

    For example, say we have 70 Winelib/C tests that are big
and bloated and we want to fix that.  We could define a
new make target, say a '.test.so'.  We could modify winetest
to be able to load a .test.so file and call main() with
the right parameters and suddenly all of our bloat is gone.
We'd just need a rule to build a .test.so file from
a .test.c file, and then we'd need to tweak our foo.test
file to add a new command 'so_test=foo.test.so' instead
of 'invoke=foo' and we're done.  We could probably even
eliminate the .spec file (only for console apps, obviously).

    And another thing, what if I have an app that creates a 
graphical image and then uses Gimp or some other scripting 
process to take screen shots and then compares them to a known
reference?  In a makefile only solution, it'd be a pain
to add a target to call some fancy script to perform
a regression test.  This case is already handled by
the .test file.

    And don't forget the Julian fries!  The very best
feature of the framework! <g>

    Jer