To error out or to skip tests?

[Nate Gallaher]

It sounds to me like the root problem is not that tests were being 
skipped, but that developers didn't know that the test machines had no 
sound configured. Perhaps simply adding -nosound to the testbed names 
would clear up the issue? Then developers interested in sound can look 
for testbed machines that don't have that tag and that don't skip their 
tests?

Taking it to the next level I think would involve some sort of "expected 
configuration" file, setup for each test machine, which you feed to the 
test framework. The framework then flags particular skipped tests as 
errors for that expected configuration. Then someone would have to 
maintain this and make sure there's adequate coverage.  This seems like 
a lot of work, and potentially a lot of spaghetti, which makes me think 
#1 is the better option for now.

Regards,
~Nate

Joerg-Cyril.Hoehle at t-systems.com wrote:
> Hi,
>
> I apologize for the numerous occurrances of red color on test.winehq
> due to my new winmm MCI tests.  That was not expected.
>
> However, turning something green is not a goal in itself.  For some
> time now I wanted to query wine-devel about this topic, independently
> of my patches, so here it goes.
>
> An analysis reveals that my tests implement a detector of
> vmware (virtualbox etc.) default installs, i.e. MS-Windows
> systems that have not been configured with sound.
>
> No real user's machines would be like that
> (except for a w95 box I tried with a broken soundcard).
>
> The revealing line is
>   
>> Tests skipped: Please install audio driver. Tests will fail (18 errors)
>>     
> This goes hand in hand with "5 tests, 0 errors, 1 skip" in winmm:wave.
> Exactly those systems skipping the wave tests produce 18-19 MCI errors.
>
> Failure to flag an error in such cases has known drawbacks:
> - For several month, Austin did not notice that his BSD boxes had lost
>   sound.  wave.ok simply reports "5 tests ok, 1 skipped" in joyfull
>   green, whereas it executed many tests half a year ago.
>
> - All those vmware boxes configured without sound and thus skipping
>   all but 5 wave.ok tests have for monthes *not* provided valuable
>   feedback to Wine.  So they were actually *not* executing tests, when
>   they could have.
>   It's only when Paul tested one of my tests that he realized that
>   sound was not configured on one of his vmware boxes, promptly
>   configured the sound device, and the tests passed.
>
> The picture on test.winehq.org is a disaster: None of the w95, w98, nt4,
> 2k, 2k3, Vista, 2k8, w7 machines have performed winmm:wave tests!
> They all have no sound configured.  Mostly useless.
>
> Therefore, I believe it's sometimes better to flag an error than to
> calmly skip something.
>
> Last year already, submitting the ICCVID patches, I was thinking:
> "iccvid.dll may not part of every installation, but we want feedback
> so we should compare behaviour with MS-Windows systems equipped with
> that library and ask people to install it if not present".
> Likewise, Wine should compare with systems similarly equipped: d3d8,
> d3d9, dplay, dmusic etc.
>
> Perhaps an easy change on test.winehq would be to augment the blue
> color indicator of skipped tests to the full rectangle surface instead
> of the current green?
>
> Of course, I'll change my tests to flag a single error on systems
> without sound.  -- Or no error at all, if that's wine-devel consensus.
>
> What do you think?
>      Jörg Höhle.
> There's only few real MCI test errors I saw on test.winehq: the system
> "fg_winxp-ie8" that yields 1 error instead of 18:
> mci.c:391: Test failed: not enough time elapsed 80ms -- a flakey test.
> And on the ME systems:
> mci.c:603: Test failed: mci re-auto-open status mode returned error: 275
> mci.c:612: Test failed: mci auto-open pause returned error: 275
> Luckily, there are a few machines that pass all tests.
>
>
>