To error out or to skip tests?

[Kai Blin]

On Friday 06 November 2009 13:21:45 Joerg-Cyril.Hoehle at t-systems.com wrote:

> The picture on test.winehq.org is a disaster: None of the w95, w98, nt4,
> 2k, 2k3, Vista, 2k8, w7 machines have performed winmm:wave tests!
> They all have no sound configured.  Mostly useless.

By extension of your logic, all results from Windows machines < Vista and from 
most Wine boxes are useless for ws2_32 tests, as they all don't have IPv6 
configured so the IPv6 tests skip. To pick up on Nate's idea, I'd request that 
everybody adds a -ipv4 or -ipv6 suffix to the names of the test reports.

Additionally, no single test machine runs as part of a windows AD or even NT4 
domain. Thus, a couple of advapi32 tests are skipped, making them worthless. 
Perhaps people should add a -nodomain suffix to their test report names.

If I make all of those tests error out, I'll be drowning real test failures in 
the noise of configuration issues. I think the same thing applies to your 
sound tests. Btw, just ask the d3d folks about their woes with (almost) 
everybody running their windows-based tests on virtual machines, making a lot 
of the d3d tests useless. Not to mention that there's many people who run 
hardware with really lousy graphics drivers. Having to skip tests because the 
system's configuration does not support running them is just something we need 
to deal with.

Kai

PS: I was joking about the suffixes. Just to make that clear. I don't want to 
go having to call my test boxes win2k3-kb-vb-nosound-domain-noadmin-ipv6-ie7-
bluebg-classictheme-en_IE-nodotnet

-- 
Kai Blin
WorldForge developer  http://www.worldforge.org/
Wine developer        http://wiki.winehq.org/KaiBlin
Samba team member     http://www.samba.org/samba/team/
--
Will code for cotton.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20091109/1cde8524/attachment.pgp>