To error out or to skip tests?
Paul Vriens
paul.vriens.wine at gmail.com
Mon Nov 9 04:47:20 CST 2009
On 11/09/2009 11:27 AM, Joerg-Cyril.Hoehle at t-systems.com wrote:
> Reece Dunn wrote:
>
>>> The question remains:
>>> a) turn sound tests red on machines without sound (with 1
>> error only);
>>> b) turn them blue (clearly marking skipped tests)
>>> c) turn them green -- status quo (for dsound.ok and wave.ok).
>
>> 310 ok(!err,"mci open waveaudio!tempfile.wav returned error: %d\n", err);
>> 311 if(err) {
>> 312 skip("tempfile.wav was not found (not saved), skipping\n");
>> Here, you are doing both (generating an error and skipping)!
>
> On purpose. I wanted an error flagged to point people's attention to the
> issue of machines without sound, so that it can be discussed.
>
>> I think that it is safe to say that if there is no audio driver
>> present, all subsequent mci tests will fail and should also be skipped
>> (ideally with just the single warning, e.g. by returning FALSE from
>> that function and wrapping the other tests in an if).
>
> My current line of thought is that broken and skip are somewhat *bad*
> things, because by not performing tests, we don't learn anything about
> the behaviour in "unusual" circumstances.
>
There are so many variables on systems that these were introduced some
time ago. The general feeling is that we try to have succeeding tests on
a many boxes as possible. Currently we are already having a hard time
achieving that on clean/correct (so we think) boxes. Once the majority
of these clean boxes are showing up green we can start thinking of
adding tests for broken configurations.
You are of course free to have your tests succeed for every possible
combination out there, but the key word is 'succeed' here.
So if you find that we have test failures for a non-admin user with a
Transylvanian locale with no soundcard it still should show up green
(with or without the blue borders).
> What's the purpose of skip()? Nothing but generate blue colour in
> test.winehq (use trace(); return; if all you need is print something
> and avoid other tests). What's the purpose of the blue colour?
> - A call for action?
> + "please run again in the english locale so I can perform locale-dependent tests";
> + "please install sound so that you'll know whether it works on BSD"
> + "please install Gecko so I can perform more tests"
> - ...?
In my opinion the main purpose for skip() is to not run tests for
legitimate reasons. If the tests are written so that they can only be
run (aka succeed) when a sound card is present you skip() these tests on
systems without a soundcard. skip() should not be used to get everything
green (the same is in theory true for broken()).
A good example of using skip() is when functions are just not available
on a platform (like most W-calls on 9x boxes). We also have some tests
that will only succeed when the user is an administrator. It's perfectly
valid to use skip() if we find the user isn't.
Broken() is a little bit different with that respect that we usually
mark platforms/boxes as broken() when 99% (or at least the majority) of
other configurations succeed these tests.
>
> Regards,
> Jörg Höhle.
--
Cheers,
Paul.
More information about the wine-devel
mailing list