Babystep: Testing framework

[Alexandre Julliard]

Jeremy White <jwhite at codeweavers.com> writes:

>     The normal behaviour of the tests is that they are
> oblidged to provide an 0 exit code on success; I think that
> is an excellent standard.
>     However, having the diff feature allows us to more
> rapdily adapt existing programs to become tests.  
> Since it's done (and it's trivial code), I don't see
> the harm in leaving in the feature.  We can hide it/discourage
> in in the (as yet unwritten) doco if you like.

The problem is that it's hard to support both types of tests from the
makefile, so it would force tests without output to generate dummy
output and reference files. I think it's better to make the default be
no output and have tests that need output diffs do the extra work.

>     The entire samples directory that I submitted consisted 
> of 2,705 bytes.  For all CVS checkouts and 'normal' Wine use, 
> there will be an insignificant penalty.

"Normal" Wine use should definitely include running regression
tests. I don't want to be the only one to run them.

>     For example, say we have 70 Winelib/C tests that are big
> and bloated and we want to fix that.  We could define a
> new make target, say a '.test.so'.  We could modify winetest
> to be able to load a .test.so file and call main() with
> the right parameters and suddenly all of our bloat is gone.
> We'd just need a rule to build a .test.so file from
> a .test.c file, and then we'd need to tweak our foo.test
> file to add a new command 'so_test=foo.test.so' instead
> of 'invoke=foo' and we're done.  We could probably even
> eliminate the .spec file (only for console apps, obviously).

But without your .test file it's even easier to do, all you have to do
is to hack the makefile. Plus you can get the dependencies right (how
can make guess what file your .test is referencing?)

>     And another thing, what if I have an app that creates a 
> graphical image and then uses Gimp or some other scripting 
> process to take screen shots and then compares them to a known
> reference?  In a makefile only solution, it'd be a pain
> to add a target to call some fancy script to perform
> a regression test.  This case is already handled by
> the .test file.

You can do that from a Perl test too, but I don't think this should be
encouraged. Tests should not depend on external programs any more than
strictly necessary, otherwise people will soon need a complete Linux
distribution on their Windows machine to run our tests.

-- 
Alexandre Julliard
julliard at winehq.com