Wine conformance testing on native platforms

[Dimitrie O. Paun]

On December 11, 2002 12:21 pm, Luke Stras wrote:
> Dimi, I see that you've added me as a "volunteer" for running the
> conformance tests on NT.  Gee, thanks :).

You're welcome! :)

> In any case, I don't mind doing this.  However, I have a question : what
> should I do with the test results?  Posting them to wine-devel every
> time a test changes seems somewhat high-bandwidth...

Well, theoretically these tests should not fail. So once we fix them,
you should post a message to wine-devel if and only if you notice a 
failure.

> What we (i.e., the testers) could do is to keep track of what changes
> between successive test runs, I guess, and we could just post a summary
> of the changes.

Yes, that would be useful to track progress until we get all tests fixed.
diff(1) is your friend. But once the tests are fixed, you only need to
post if you notice any failure, as explained above.

> This brings up another point: these test results should be put up
> somewhere on the Web for everyone to see.  Might I suggest putting them
> up on the Wine Janitorial Projects page?  For every platform, just put
> in a link to "Most recent test results", say.

I've added a link there to the binaries posted by Francois. Would you
prefer something else?

> One last thing: is there any way to be automatically notified whenever
> the available test suite changes?

I suggested it, but someone has to
  1. Write a script that build the tests, creates an archive, compares the
     result with the previous build, and if different, emails the guys on
     the testing list
  2. We need a volunteer that runs the script from time to time
     (say sets up a cron job on a box that does 
	cd "$WINEROOT" && cvs up && tools/update-tests
      or something like this)

I've also included on the 'Fun Projects' page an idea for a nice test
harness (a la JUnit). Check it out -- perfect project for a newbie with
some Win32 programming experience.

-- 
Dimi.