Enhancing winetest infrastructure [WAS: Wineconf agenda]

Jakob Eriksson jakov at vmlinux.org
Thu Mar 17 09:52:54 CST 2005

Paul Millar wrote:

> On Thursday 17 March 2005 11:54, Jakob Eriksson wrote:
>> Maybe one could script something up so that a commit to
>> tests CVS would not be *possible* without a confirmed test
>> pass on Windows 95, NT, 2000 and XP.
>> That'd be the best thing since sliced bread.
> CVS supports doing some server-side validation tests before accepting
> a commit, so it could be done; but I don't think this would be
> "nice". CVS is a mechanism for sharing code: its not really a
> testing framework.

I didn't mean exactly on the CVS level. When Alexandre commits any
patch, he first checks that the code still passes regression tests.

I want something similar for the test patches. Maybe like this:
the testing dispatcher signs a working patch* with GPG.
(Or no GPG. Just set a flag somewhere. Details are not important.)

Alexandre will see this flag when saving a patch from
wine-patches at winehq and know that the patch is OK as far as the
test grid is concerned.

It could be as non-intrusive as this: the test dispatcher monitors
the wine-patches at winehq for patches. As soon as it sees a patch it
recognises and knows it has tested, it sends a mail to wine-patches
akin to:

The patch with CRC32 so-and-so posted by him or her, named so-and-so
is hereby verified by me, the Wine Regression Grid Tester. **


The patch with CRC32 so-and-so posted by him or her, named so-and-so
failed under the following versions of Windows; bla bla blah, with
the following error message:
blah blah bla some more.
Truthfully yours, the Wine Regression Grid Tester. **

Then it's up to Alexandre if he wants to commit a test which the
grid tester has rejected, or for which there is no confirmation.

If you don't like the idea of a program spamming wine-patches, it
could be separate list, or a webpage with a copy of wine-patches,
with different messages colour-codes updated as they get tested
by the grid tester.

* A working patch is a patch that has been tested and found
working on Win 95, 98, ME, NT4, 2000 and XP.

** Could we call it WineGrind? :-)

> For the testing framework, I'd say what we have just now is fine. It
> lives outside (and on top of) CVS. Having broken tests is OK,
> provided they're fixed within a suitable time-scale [*].

Actually, I think having broken tests is not OK. It not only goes
against my zealotry for Extreme Programming, it's also very annoying
when I have _no_clue_ how to fix a broken test and the author is
missing or don't want to touch the code with a ten foot stick.

> (just my 2c-worth again)
> Cheers,
> Paul
> [*] -- Of course, what is the "suitable time-scale"? who's willing to
> make sure things get fixed?

Exactly, I feel rage everytime I see those red and yellow boxes at
http://test.winehq.org/data/   ;-)


More information about the wine-devel mailing list