To error out or to skip tests?

[Joerg-Cyril.Hoehle at t-systems.com]

Reece,

I think we reached agreement on:
a) Tests should "usually" succeed.
b) machines without sound are usual (e.g. in virtual server env.).
c) I'll modify the MCI tests to succeed on machines without sound.
But for some other points, I've trouble following you.


>> The relevant question becomes: when is a configuration unusual? When
>> can it be ignored for the purpose of Wine and when is it relevant?
>Answer: no configuration is unusual.
Huh?
>The point is, if the machine has a broken soundcard (or in Wine does
>not have an available sound driver, or is broken due to PulseAudio on
>Ubuntu), the tests should still pass (or be skipped) as is
>appropriate.

This seems wrong (in general) to me, even though Paul somehow
expresses an opinion similar to yours.  I say you cannot write a
program meant to run in a completely unknown environment, doing
comprehensive tests and not stumbling upon strange behaviour.
The program is specifically designed to detect and flag deviant behaviour.
All that can be achieved in a program is to be prepared for a known
set of scenarios (a broken soundcard is such a scenerio), aka. use-case.
Outside these known use-cases, the tests throw an error.  Otherwise
there's no point in ok() at all.

I agree that for the known use-cases (these explicitly include
configurations without sound), the tests should succeed.

What can be done is what you say:
>If we find that other situations need correcting (due to
>failing tests), they can be looked at then.
... then decide whether the cause of the failing test is "unusual" or not.


I feel that I shall stop discussing that issue.  If the consensus
is good enough, we need not take this further to eventually discover
that the sole disagreement is on the exact meaning of some English words.


>It also seems to me that there should be some tests that test opening
>an mci stream on a file that we know not to exist (e.g.
>this-file-does-not-exist.wav) so that those failure cases can be
>properly documented.
Good idea, esp. that w95 returns different error codes for foo.wav and bar\foo.wav!


>The point I am making is that:
>  1.  The error and skip statements are saying the same thing;
>  2.  But the skip message is misleading.
Indeed, sorry for that.
There's a dependency.  The tests depend on record working to generate the
file to be played, instead of including one in git.
So they fail to test machines that can play but not record.
Obviously I'll have to do something else in the yet to be written MIDI tests.

BTW, this raises the question whether Wine should include a tiny wave file
for completeness.  Invoking the MCI Sound command always plays some sound.


>If I have counted the offsets correctly 275 is
>MCIERR_DEVICE_NOT_READY, *not* that tempfile.wav was not found.
Wrong.  275 - 256 is 19, which you look up in mmsystem.h:
#define MCIERR_FILE_NOT_FOUND   (MCIERR_BASE + 19)


>   default:
>      ok(!err, "mciSendString('open') failed, expected NOERROR, got
>%s", dbg_mcierr(err));
>   }
In the default case, if (err) you must return (with or without skip message).
You cannot continue if open fails.  So it summarizes to:
if (err) {
    /* list of situations that were analysed as "usual": */
    ok(err=FILE_NOT_FOUND || err==MCIERR_CANNOT_LOAD_DRIVER || err==xyz ...);
   skip("Cannot test playback, cause %s",dbg_mcierr(err));
   return
}

>   case MCIERR_CANNOT_LOAD_DRIVER:
>   2.  sound is not present on the system.
Perhaps there's a misconception? That's not how MS-Windows systems without
sound drivers present themselves.  Ge's, Paul's and my father in law's
machines all include the MCI (and waveaudio.dll etc.).  It's only at
WAVE_Open time that failure is detected, e.g. Cue, Record or Play, and
the error is MCIERR_WAVE_INPUTSUNSUITABLE for instance.
(That's not entirely true, it does not explain the setformat pcm failure).
I.e. no mciwave present at all so far is an "unusual" situation, and generates an error.

Regards,
	Jörg Höhle.