Call for winetest ing

[Paul Vriens]

Hi Jeremy,

Jeremy White wrote:
> Hi Paul,
> 
>> Something we also have with the current tests is that the source files actually
>> point to the correct version in CVS. I now see some strange links to refs on
>> http://test.winehq.org/data/20080205/.
> 
> Well, I'm afraid I'd argue that the current tests should be updated to point
> to git...
> 
>> Even though the HEAD sha1 looks better than the date for a build-id it will
>> again introduce a lot of directories with only a few tests in there.
> 
> Is that really true?
> 
> Alexandre tends to touch the tree once a day, usually all in one batch.
> If folks kick off winetest right before the leave for the day, wouldn't
> the odds be that everyone would have the same HEAD sha1?  (I guess that requires
> people to run winetest from a clean, winehq only tree).
> 
> And even if it is true, isn't it more logically correct?

Ok, so even if HEAD tends to be the same doesn't that mean the same test doesn't 
get run on Windows versions? With a common source (and I agree that there 
probably should be more) it better to compare.

If you build both the winetest your self and run the tests with that same 
winetest the objectivity goes away, not?

> 
> Right now, I could download a winetest.exe and use
> Wine-0.9.1 to run it, and it would be grouped in the same bucket as someone running
> off their own badly hacked wine tip tree.  Doesn't that create misleading data?
>

That's true and I don't see an easy solution to that. There are loads of 
variables that influence the outcome of test as mentioned in this email but also 
in the rest of the email-chain.
We already see failures with tests on Wine that were not present on AJ's box.

> However, I agree that if we changed that now it would be a mess; we'd need to change
> the script that presents the directories to group them by date so as to present the
> more recent sha1 keys.
> 
>> I don't see a problem with running the winetest executable under Wine the same 
>> as I do under all kinds of Windows versions.
> 
> Perhaps you're right.  Occam would suggest you are.
> 
> But it sticks in my craw.  I have a source tree; I have
> built the files; I can run make test and it fails badly.  I want to tell the world
> about my failures now (/me stamps his little feet  <grin>).
> 
> Getting a .exe from you requires you to have reliably built winetest.exe, and does not
> allow me to test the tip *now*.  Further, if there are any discrepancies introduced
> by the way my system builds the test .exe files compared to yours, running your winetest.exe won't
> catch them.  (I realize that may be pure rationalization).
>

'You' in this case is Paul Millar and not this Paul ;-).

> I think to some extent we are at different purposes, which may explain a
> reason to change.
> 
> I think winetest has long been used to test the tests themselves, and to identify
> the issues with the tests and fix them.  That is, the variable is the tests, and the underlying
> OS (i.e. Windows) is essentially a constant.
> 
> But my purpose requires 2 variables - the tests changing as needed, but also the
> underlying OS - Wine - changing rapidly as well.
> 
> Cheers,
> 
> Jeremy
> 

-- 
Cheers,

Paul.