Testing edge cases and undocumented behavior

[Joerg-Cyril.Hoehle at t-systems.com]

Scott Ritchie wrote:
>There was a bit of a philosophical discussion on #winehackers about the
>merits of creating tests for functions that might be testing undefined
>or unimportant behavior.  Windows behaves one way, we behave another,
>the tests measure this delta, but it's unknown if this will actually
>improve a real world application.

Vincent Povirk wrote an excellent reply.  I hope somebody will turn
that into a Wiki page.

>More broadly, should we resist any change without a particular
>(real-world) application in hand that needs it?
This is stupid because you're always late.  Like a fireman.
a) Somebody who wrote the bug report is waiting for the bug to be fixed ASAP.
b) It takes a lot of effort to analyse huge and cryptic log files to
   find out what the particular delta is.

>Or should we err on the side of testable behavior,
What you test is what you know.  It's not erring.
If you don't know what the behaviour should be, you can't add a
FIXME pointing out that Wine deviates from native behaviour.

When I asked about testing depth w.r.t. the MCI a few years ago, Paul
Vriens advised to test to the bones.  IMHO that's good advice.

I recommend beginners to start writing tests *first*.  Doing so, you
learn a lot about the component Wine purports to mimic.
You can then even start to predict failures!

Somehow I believe I'm a very untypical Wine contributor.  I'm not bug
driven.  Quite to the contrary, my thesis is that once you've tested a
component to the bones *and* replicated the observable behaviour,
there will be few opportunities left for people to report bugs.

The basic alternative is that if you have free time now, you can
either try to understand obscure logs of some component you're not
exactly familiar with and more or less well documented, perhaps
single-step your way through the debugger, *OR* you can spend time
shedding light on said component via tests, mostly working with logs
you control because you're working with full source to your test and
the Wine code.

That's what I was doing with the audio parts of the MCI before WinMM
was rewritten to use mmdevapi.  Now that's what I'm doing with mmdevapi.
After that, should I still have the energy, I'll look at WinMM audio.

Other benefit:
 - Tests get executed daily whereas an application X can be made to
   work in wine-X yet break again in X+1 until somebody notices.

Regards,
	Jörg Höhle