Enhancing winetest infrastructure [WAS: Wineconf agenda]

[Jakob Eriksson]

Paul Millar wrote:

> On Thursday 17 March 2005 11:54, Jakob Eriksson wrote:
>
>> Maybe one could script something up so that a commit to
>> tests CVS would not be *possible* without a confirmed test
>> pass on Windows 95, NT, 2000 and XP.
>> That'd be the best thing since sliced bread.
>
>
> CVS supports doing some server-side validation tests before accepting
> a commit, so it could be done; but I don't think this would be
> "nice". CVS is a mechanism for sharing code: its not really a
> testing framework.



I didn't mean exactly on the CVS level. When Alexandre commits any
patch, he first checks that the code still passes regression tests.

I want something similar for the test patches. Maybe like this:
the testing dispatcher signs a working patch* with GPG.
(Or no GPG. Just set a flag somewhere. Details are not important.)

Alexandre will see this flag when saving a patch from
wine-patches at winehq and know that the patch is OK as far as the
test grid is concerned.

It could be as non-intrusive as this: the test dispatcher monitors
the wine-patches at winehq for patches. As soon as it sees a patch it
recognises and knows it has tested, it sends a mail to wine-patches
akin to:

The patch with CRC32 so-and-so posted by him or her, named so-and-so
is hereby verified by me, the Wine Regression Grid Tester. **

or

The patch with CRC32 so-and-so posted by him or her, named so-and-so
failed under the following versions of Windows; bla bla blah, with
the following error message:
blah blah bla some more.
Truthfully yours, the Wine Regression Grid Tester. **


Then it's up to Alexandre if he wants to commit a test which the
grid tester has rejected, or for which there is no confirmation.



If you don't like the idea of a program spamming wine-patches, it
could be separate list, or a webpage with a copy of wine-patches,
with different messages colour-codes updated as they get tested
by the grid tester.



* A working patch is a patch that has been tested and found
working on Win 95, 98, ME, NT4, 2000 and XP.


** Could we call it WineGrind? :-)




>
> For the testing framework, I'd say what we have just now is fine. It
> lives outside (and on top of) CVS. Having broken tests is OK,
> provided they're fixed within a suitable time-scale [*].


Actually, I think having broken tests is not OK. It not only goes
against my zealotry for Extreme Programming, it's also very annoying
when I have _no_clue_ how to fix a broken test and the author is
missing or don't want to touch the code with a ten foot stick.



>
> (just my 2c-worth again)
>
> Cheers,
>
> Paul
>
> [*] -- Of course, what is the "suitable time-scale"? who's willing to
> make sure things get fixed?


Exactly, I feel rage everytime I see those red and yellow boxes at
http://test.winehq.org/data/   ;-)


regards,
Jakob