Testing edge cases and undocumented behavior

[Michael Stefaniuc]

On 08/26/2011 12:45 PM, Joerg-Cyril.Hoehle at t-systems.com wrote:
> Scott Ritchie wrote:
>> There was a bit of a philosophical discussion on #winehackers about the
>> merits of creating tests for functions that might be testing undefined
>> or unimportant behavior.  Windows behaves one way, we behave another,
>> the tests measure this delta, but it's unknown if this will actually
>> improve a real world application.
> 
> Vincent Povirk wrote an excellent reply.  I hope somebody will turn
> that into a Wiki page.
> 
>> More broadly, should we resist any change without a particular
>> (real-world) application in hand that needs it?
> This is stupid because you're always late.  Like a fireman.
> a) Somebody who wrote the bug report is waiting for the bug to be fixed ASAP.
> b) It takes a lot of effort to analyse huge and cryptic log files to
>    find out what the particular delta is.
> 
>> Or should we err on the side of testable behavior,
> What you test is what you know.  It's not erring.
> If you don't know what the behaviour should be, you can't add a
> FIXME pointing out that Wine deviates from native behaviour.
> 
> When I asked about testing depth w.r.t. the MCI a few years ago, Paul
> Vriens advised to test to the bones.  IMHO that's good advice.
It is a good advice. But that doesn't means one has to submit tests that
don't make sense.

> I recommend beginners to start writing tests *first*.  Doing so, you
> learn a lot about the component Wine purports to mimic.
> You can then even start to predict failures!
> 
> Somehow I believe I'm a very untypical Wine contributor.  I'm not bug
I'm not bug driven either,

> driven.  Quite to the contrary, my thesis is that once you've tested a
> component to the bones *and* replicated the observable behaviour,
but here I heartily disagree. You don't test nor replicate all the
observable behavior; that is way way too broad. An extreme case would be
to try to test how much time/CPU cycles a function call takes and trying
to replicate that in Wine. In general that doesn't matter but is is
observable and testable.

What you do want to test is:
- the de jure API (as documented), and
- the de facto API (as used by applications and thus kept "broken"
across Windows version).

So if an application relies on the fact that a function call will crash
with an invalid combination of arguments than Wine will have to crash
too but it doesn't matter if the segfault happens at address 0x00000002
or 0x00000001.

> there will be few opportunities left for people to report bugs.
You sub-estimate the ingenuity of application developers to misuse the
API. Or you found a way to solve the halting problem ;)

> The basic alternative is that if you have free time now, you can
> either try to understand obscure logs of some component you're not
> exactly familiar with and more or less well documented, perhaps
> single-step your way through the debugger, *OR* you can spend time
> shedding light on said component via tests, mostly working with logs
> you control because you're working with full source to your test and
> the Wine code.
It is not an either or but a "do both", for the project in general that
is. People are free to scratch their itch as long as they don't make the
code a mess.

bye
     michael