Babystep: Testing framework

[Jeremy White]

> I can't say I like your *.test files; I think this should be taken
> care of by the makefile directly. 


     I started down this road, but then I ended up needing
PERL_TESTS and INVOKE_TESTS in the Makefile, and IMHO,
that was uglier than the *.test file.
     This may simply be a lack of Makefile moxy on my part;
the point where I threw up my hands in disgust was when
I was requiring my Winelib/C tests to be compiled into
a 'foo.itest' so that I could do a .itest.test.out dependency
rule.


> Also it seems the consensus is that
> we should simply check the exit status of the test, not compare the
> test output.


     Sorry, this wasn't clear.  That is the default behavior.
I preserved the pattern and reference checking as an optional
addition.




> 
> 
>>Flaws:
>>     1.  It has no provision for use on Windows.
>>         IMO the right way to fix this is to
>>         go through the effort of making Wine's
>>         ./configure script be intelligent enough
>>         to build 'Winelib' programs, so that
>>         in Windows a 'make test' should just work.
>>
> 
> I'm not sure about that; I think it would be preferable to have a
> simple, self-contained environment. Running configure would require a
> lot of stuff to be installed on the Windows side.


Well, I was sufficiently impressed with the Cygwin install
to think it wouldn't be that bad; if nothing else, I think
we should try to have both.


> 
> 
>>     2.  I really don't like that a C/Winelib
>>         test requires its own directory
>>         (see samples/sample3).  However, AFAICT,
>>         that was the only way to create a simple
>>         and clean build environment for the Winelib app.
>>
> 
> That's the problem with C tests. You cannot have one executable for
> each test, at least not with Winelib, so we need a way to build
> multiple tests inside a single executable. Not very hard, but there's
> a bit of work involved.


I would argue, that except perhaps for your objection to
my *.test format, that the framework as I have proposed would
be a good start.  We can refine the Winelib stuff later.

It seems to me that the alternate is to spend another five
weeks debating possible frameworks, and ending with none implemented.

Jer