sandbox for regression tests

[Francois Gouget]

On 23 Mar 2002, Geoffrey Hausheer wrote:

> Hello everyone,
> I was considering writing a few regression tests for wine,
> and after reading Francois' presentation, I thought

   I'm glad my presentation has inspired you :-)

[...]
> But I don't see anything in the framework that lets me do
> this portably on all systems.

   To portably create files and directories on all system just use
CreateFile & co.


> In wine, I can't gaurantee
> that the test directory (which I could probably write to)
> is mapped to a Windows drive.  On all platformes, I don't
> see how I can write to an arbitrary location on C:, since
> it may  be unwriteable, and even if it's not, people
> probably don't want extraneous wine stuff littered around
> their drive when something goes wrong.

   I believe most file tests don't need to write to arbitrary locations
on the C:\ drive. For such tests, the best is probably to change the
current directory to $Temp (should be defined on Windows too) and create
files and directories there.
   Some other tests will need a writable c:\ drive... or other. Maybe
tests should check a WINETEST_DRIVE environment variable.

   In any case, tests should cleanup after themselves, i.e. call
DeleteFile after they are done with a file, and be able to cope in case
they were abruptly terminated, i.e. not fail if the file already exists,
or Delete it before hand.


   We may be able to add some infrastructure to handle this but doing so
is not easy because it is full of compromises:
 * we want the infrastructure to be simple. Otherwise this makes writing
tests complex which is bad.
 * a simple infrastructure is likely to only handle simple cases well.
But simple cases are those where we need the infrastructure the least.
So the infrastructure would have to be really easy to make even these
cases even easier.
 * to handle the more complex cases the infrastructure woul dhave to be
more complex. And that's bad as per point 1.
 * if each test does things its own way, things will be done slightly
differently each time. This is actually good if it means more coverage
of APIs. It's bad if it becomes a maintenance nightmare or leads to too
many poorly written tests.


   To summarise, I think it's not clear at this time exactly what the
infrastructure should do for testing files. I believe the best thing to
do is to write a few of the file tests, including some of the hard ones
like those that convert paths. Then, when we have half a dozen or more
we can look back and see if something can be done to make them simpler
(for the next hundred ;-).



--
Francois Gouget         fgouget at free.fr        http://fgouget.free.fr/
                     Avoid the Gates of Hell - use Linux.