To error out or to skip tests?

[Joerg-Cyril.Hoehle at t-systems.com]

Hi,

I apologize for the numerous occurrances of red color on test.winehq
due to my new winmm MCI tests.  That was not expected.

However, turning something green is not a goal in itself.  For some
time now I wanted to query wine-devel about this topic, independently
of my patches, so here it goes.

An analysis reveals that my tests implement a detector of
vmware (virtualbox etc.) default installs, i.e. MS-Windows
systems that have not been configured with sound.

No real user's machines would be like that
(except for a w95 box I tried with a broken soundcard).

The revealing line is
> Tests skipped: Please install audio driver. Tests will fail (18 errors)
This goes hand in hand with "5 tests, 0 errors, 1 skip" in winmm:wave.
Exactly those systems skipping the wave tests produce 18-19 MCI errors.

Failure to flag an error in such cases has known drawbacks:
- For several month, Austin did not notice that his BSD boxes had lost
  sound.  wave.ok simply reports "5 tests ok, 1 skipped" in joyfull
  green, whereas it executed many tests half a year ago.

- All those vmware boxes configured without sound and thus skipping
  all but 5 wave.ok tests have for monthes *not* provided valuable
  feedback to Wine.  So they were actually *not* executing tests, when
  they could have.
  It's only when Paul tested one of my tests that he realized that
  sound was not configured on one of his vmware boxes, promptly
  configured the sound device, and the tests passed.

The picture on test.winehq.org is a disaster: None of the w95, w98, nt4,
2k, 2k3, Vista, 2k8, w7 machines have performed winmm:wave tests!
They all have no sound configured.  Mostly useless.

Therefore, I believe it's sometimes better to flag an error than to
calmly skip something.

Last year already, submitting the ICCVID patches, I was thinking:
"iccvid.dll may not part of every installation, but we want feedback
so we should compare behaviour with MS-Windows systems equipped with
that library and ask people to install it if not present".
Likewise, Wine should compare with systems similarly equipped: d3d8,
d3d9, dplay, dmusic etc.

Perhaps an easy change on test.winehq would be to augment the blue
color indicator of skipped tests to the full rectangle surface instead
of the current green?

Of course, I'll change my tests to flag a single error on systems
without sound.  -- Or no error at all, if that's wine-devel consensus.

What do you think?
     Jörg Höhle.
There's only few real MCI test errors I saw on test.winehq: the system
"fg_winxp-ie8" that yields 1 error instead of 18:
mci.c:391: Test failed: not enough time elapsed 80ms -- a flakey test.
And on the ME systems:
mci.c:603: Test failed: mci re-auto-open status mode returned error: 275
mci.c:612: Test failed: mci auto-open pause returned error: 275
Luckily, there are a few machines that pass all tests.