Babystep: Testing framework

Jeremy White jwhite at codeweavers.com
Wed Jan 9 22:56:19 CST 2002


Alexandre Julliard wrote:

>Jeremy White <jwhite at codeweavers.com> writes:
>
>>    The normal behaviour of the tests is that they are
>>oblidged to provide an 0 exit code on success; I think that
>>is an excellent standard.
>>    However, having the diff feature allows us to more
>>rapdily adapt existing programs to become tests.  
>>Since it's done (and it's trivial code), I don't see
>>the harm in leaving in the feature.  We can hide it/discourage
>>in in the (as yet unwritten) doco if you like.
>>
>
>The problem is that it's hard to support both types of tests from the
>makefile, so it would force tests without output to generate dummy
>output and reference files. I think it's better to make the default be
>no output and have tests that need output diffs do the extra work.
>
Um...the way I constructed it, there is no additional effort required
in the Makefile whatsoever.  Now, if you get rid of my .test file,
yes, then, doing this all in the Makefile is a pain in the neck.


>"Normal" Wine use should definitely include running regression
>tests. I don't want to be the only one to run them.
>
Point taken; but I still believe that this is an issue not relevant to
my proposed patch, but is a problem with using C tests as a whole.
A solution to this problem will be just as appropriate with
or without my patch.

>
>But without your .test file it's even easier to do, all you have to do
>is to hack the makefile. Plus you can get the dependencies right (how
>can make guess what file your .test is referencing?)
>
If the goal is to make it easy for a newcomer to add a test, then requiring
them to hack the Makefile is going to be a barrier.  Asking them to
copy and cut/paste a 'samples/foo.test' file will not be.

I do concede the point that my .test file hides dependencies, and
a Makefile only approach can do that better.

>You can do that from a Perl test too, but I don't think this should be
>encouraged. Tests should not depend on external programs any more than
>strictly necessary, otherwise people will soon need a complete Linux
>distribution on their Windows machine to run our tests.
>

I threw that out as an example, not necessarily a serious case.  
I think the bottom line question is whether or not the test scripts
should be built such that we depend on a shell script (runtest)
and a configuration file for each test (foo.test), or whether we
rely completely on the Makefiles to hold the test configuration information.

I think either choice is valid, but let me lay down the gauntlet:
replace my 'winetest/samples' directory with a Make based equivalent
that allows for the same sorts of tests, and makes it similarly obvious
to a newcomer how to add a new test.  You are better with
Make than I am; you may persuade me readily with a patch.

Jer

p.s.  If you're just rejecting the .test files because they have long 
horizontal comment
lines, we could negotiate that...<g>






More information about the wine-devel mailing list