We *really* need a development model change !

[Jeremy White]

>
>
>* its exit code
> * text output on either or both of stdout and stderr, both of which are
>normally redirected to a file called 'xxx.out'.
>
>   A test succeeds if:
> * its exit code is 0
> * and its output, 'xxx.out' matches the reference output according to
>the rules described later.
>
I think that it would be handy to use stderr for status/diagnostics, and 
only use
stdout for reference checking.  Chatty people like me get their statii,
but the tests remain clean.  Perhaps a WINETEST_DEBUG env variable
would be a good addition as well (and since Alexandre controls the commits,
my guess is it'll default to off <g>).

>Otherwise this file is either called:
> * 'xxx.ref'
> * or 'xxx.win95' or 'xxx.win98' ... if the output depends on the
>Windows version being emulated. The winever-specific file takes
>precedence over the '.ref' file, and the '.ref' file, which should
>exist, serves as a fallback.
>
I hope that the cases where this would be needed would be few enough
that we wouldn't need to build in a general purpose exception;
we can just have a mytest.win95.test and a mytest.win98.test for
the cases where it's needed.

[snipping chunks largely agreed with]

>
>Test coverage
>-------------
>
>   Each test should contain a section that looks something like:
>
I started trying to tweak Alexandre's patch to create some sample
tests, and I learned the following:
    1.  Cygwin installation has *dramatically* improved.
         Getting a full working toolchain is no longer
         a big pain in the rear end, it's actually pretty easy.

    2.  Having '.test' files be implicitly Perl scripts is too
         limiting,  IMHO.  I hate to add another format, but
         I've been toying with YAFF (yet another file format)
         so that a '.test' file describes a test, as follows:
            # Comment lines
            script=name_of_my_perl_test_script
            invoke=name_of_c_test_or_shell_script
            status=-eq 0
            pattern=.*OK
            compare=name_of_my_ref_file
 
         where one of script or invoke is required, and
          status, if given, is a test expression that $0 is compared to,
          pattern is a regexp applied to stdout, if given,
          and compare is the name of a .ref.out file to compare
          the output with.

          The default would be just a case of 'status=-eq 0'.

          The nice thing about this approach is that you can
          handle the multiple version testing just by creating
          a new reference file and a new .test file.
         

           

>In Windows:
>
>   Hmmm, not sure how that is done. Run 'winetest.exe'?
>
IMO, we should have a script that creates a 'winetest.zip',
with simple batch files to make it easy to run a single
test.

But, (as a new convert to the ease of use of Cygwin), I think
we can just stick with 'make tests' for the full deal
on Windows.

Jer