To error out or to skip tests?

[Paul Vriens]

On 11/09/2009 11:27 AM, Joerg-Cyril.Hoehle at t-systems.com wrote:
> Reece Dunn wrote:
>
>>> The question remains:
>>> a) turn sound tests red on machines without sound (with 1
>> error only);
>>> b) turn them blue (clearly marking skipped tests)
>>> c) turn them green -- status quo (for dsound.ok and wave.ok).
>
>> 310     ok(!err,"mci open waveaudio!tempfile.wav returned error: %d\n", err);
>> 311     if(err) {
>> 312         skip("tempfile.wav was not found (not saved), skipping\n");
>> Here, you are doing both (generating an error and skipping)!
>
> On purpose. I wanted an error flagged to point people's attention to the
> issue of machines without sound, so that it can be discussed.
>
>> I think that it is safe to say that if there is no audio driver
>> present, all subsequent mci tests will fail and should also be skipped
>> (ideally with just the single warning, e.g. by returning FALSE from
>> that function and wrapping the other tests in an if).
>
> My current line of thought is that broken and skip are somewhat *bad*
> things, because by not performing tests, we don't learn anything about
> the behaviour in "unusual" circumstances.
>

There are so many variables on systems that these were introduced some 
time ago. The general feeling is that we try to have succeeding tests on 
a many boxes as possible. Currently we are already having a hard time 
achieving that on clean/correct (so we think) boxes. Once the majority 
of these clean boxes are showing up green we can start thinking of 
adding tests for broken configurations.

You are of course free to have your tests succeed for every possible 
combination out there, but the key word is 'succeed' here.

So if you find that we have test failures for a non-admin user with a 
Transylvanian locale with no soundcard it still should show up green 
(with or without the blue borders).

> What's the purpose of skip()? Nothing but generate blue colour in
> test.winehq (use trace(); return; if all you need is print something
> and avoid other tests).  What's the purpose of the blue colour?
> - A call for action?
>    + "please run again in the english locale so I can perform locale-dependent tests";
>    + "please install sound so that you'll know whether it works on BSD"
>    + "please install Gecko so I can perform more tests"
> - ...?

In my opinion the main purpose for skip() is to not run tests for 
legitimate reasons. If the tests are written so that they can only be 
run (aka succeed) when a sound card is present you skip() these tests on 
systems without a soundcard. skip() should not be used to get everything 
green (the same is in theory true for broken()).

A good example of using skip() is when functions are just not available 
on a platform (like most W-calls on 9x boxes). We also have some tests 
that will only succeed when the user is an administrator. It's perfectly 
valid to use skip() if we find the user isn't.

Broken() is a little bit different with that respect that we usually 
mark platforms/boxes as broken() when 99% (or at least the majority) of 
other configurations succeed these tests.

>
> Regards,
> 	Jörg Höhle.

-- 
Cheers,

Paul.