To error out or to skip tests?

[Joerg-Cyril.Hoehle at t-systems.com]

Reece Dunn wrote:

>> The question remains:
>> a) turn sound tests red on machines without sound (with 1 
>error only);
>> b) turn them blue (clearly marking skipped tests)
>> c) turn them green -- status quo (for dsound.ok and wave.ok).

> 310     ok(!err,"mci open waveaudio!tempfile.wav returned error: %d\n", err);
> 311     if(err) {
> 312         skip("tempfile.wav was not found (not saved), skipping\n");
>Here, you are doing both (generating an error and skipping)!

On purpose. I wanted an error flagged to point people's attention to the
issue of machines without sound, so that it can be discussed.

>I think that it is safe to say that if there is no audio driver
>present, all subsequent mci tests will fail and should also be skipped
>(ideally with just the single warning, e.g. by returning FALSE from
>that function and wrapping the other tests in an if).

My current line of thought is that broken and skip are somewhat *bad*
things, because by not performing tests, we don't learn anything about
the behaviour in "unusual" circumstances.

The relevant question becomes: when is a configuration unusual? When
can it be ignored for the purpose of Wine and when is it relevant?

- I don't care that tests fail on my father in law's machine with a
  broken sound card (or rather, the card maybe ok, but somehow the
  sndblast.dll doen't manager to install it).
  Yet testing on that machine was very revealing (black box testing
  with broken configurations tells you a lot about what's inside, like
  scientists learn a lot from patients with a local brain disease).

- Ge Geldorp pointing out that some vmware engines do not support
  sound at all renders Windows without sound not unusual -- unlike
  what I initially thought.

My conclusion is that we need tests to specifically find out how
MS-Windows behaves on such machines, so that Wine can mimick it.
That might be written entirely without skip() or broken(), because
it becomes expected behaviour.


if (wave_GetNumDevs()== 0) {
   skip("no sound, no tests\n"); return;
is *not* the answer IMHO.  Turning all lights green is not a primary
goal, it's a secondary/subordinated one.


What's the purpose of skip()? Nothing but generate blue colour in
test.winehq (use trace(); return; if all you need is print something
and avoid other tests).  What's the purpose of the blue colour?
- A call for action? 
  + "please run again in the english locale so I can perform locale-dependent tests";
  + "please install sound so that you'll know whether it works on BSD"
  + "please install Gecko so I can perform more tests"
- ...?


>Shouldn't this be something like:
>      if(err == MCIERR_INVALID_FILE) {
>          skip("tempfile.wav was not found (not saved?), skipping\n");
No. If any error happens in open, we can't continue. Perhaps you mean
if (err) {
   ok(err==MCIERR_FILE_NOT_FOUND, "...\n"); /* the expected situation */
   skip("tempfile.wav was not found (not saved?), skipping\n");

>a dbg_mcierr function that was created; shouldn't that be used here
I didn't retrofit it into all messages after I wrote it. Will do so later.

Regards,
	Jörg Höhle.