To error out or to skip tests?

[Juan Lang]

Hi Joerg,

> BTW, dsound is also affected: both dsound and winmm have had little
> testing coverage despite the appealing green color.

Your concern about coverage is valid one.

> Please weight providing as much test data as possible to the Wine
> project against your lack of interest in sound issues.

I'm with Ge on this one:  I don't much care whether Wine is able to
play sound.  I care in an abstract sense, but not more than I'd like
to see green tests ;-)  Is that silly or misguided of me?  Perhaps.
On the other hand, I think there's real value in stating that "the
tests should pass on all Windows boxes," even if their configuration
is invalid or strange.  We're not there yet, but any failing test
impedes that goal.

To put it another way:  when the test failures seem intractable, it's
hard to get motivated to fix them.  For example, if there are some
tests no one understands, or that fail on systems few people run, or
if the number of test failures is simply too large, it's hard to get
motivated to chip away at the issues.  Fortunately we have a few
cheerleaders like Paul who get us motivated to work on them every now
and again.

On the other hand, when there are few test failures, there can be a
little incentive to see whether a few days' effort will produce
passing tests on at least one file, in order to see it magically
become green.  There has been for me, at least.

So, to answer your question:

> a) turn sound tests red on machines without sound (with 1 error only);
> b) turn them blue (clearly marking skipped tests)
> c) turn them green -- status quo (for dsound.ok and wave.ok).

I'm in favor of either b or c, though I lean toward c.  The reason is,
not all tests can run on all systems.  For example, you mentioned you
saw a failure on a Win95 box with a broken sound card.  If the sound
card is broken, and other test results are invalid, surely a skip is
appropriate here?  That is, this doesn't indicate an error in the
tests, and skip is appropriate.  Green vs. red tells you at a glance
what the pass rate is, though it doesn't say anything about the skip
rate.

I think what you're asking for is a separate metric for coverage and
pass rate.  I'm in favor of that.
--Juan