To error out or to skip tests?

[Paul Vriens]

On 11/09/2009 07:25 PM, Joerg-Cyril.Hoehle at t-systems.com wrote:
> Reece,
>
> I think we reached agreement on:
> a) Tests should "usually" succeed.

I didn't read that in any of the emails. Tests should always succeed. 
the broken() and skip() will help in achieving that.

>> The point is, if the machine has a broken soundcard (or in Wine does
>> not have an available sound driver, or is broken due to PulseAudio on
>> Ubuntu), the tests should still pass (or be skipped) as is
>> appropriate.
>
> This seems wrong (in general) to me, even though Paul somehow
> expresses an opinion similar to yours.  I say you cannot write a
> program meant to run in a completely unknown environment, doing
> comprehensive tests and not stumbling upon strange behaviour.

I'm now busy for example with the eventlog stuff. I create loads of 
tests and run them on my pretty clean boxes (W95 up to Win7). Only when 
they pass on all boxes I sent the patches. If test.winehq.org shows 
failures for these new tests they have to be dealt with.

I agree that not everybody has this multitude of test boxes but there 
are always people around willing to test new tests on their Windows 
boxes (real and/or virtual).

> Regards,
> 	Jörg Höhle.
>
>

-- 
Cheers,

Paul.