Testing edge cases and undocumented behavior
mstefani at redhat.com
Fri Aug 26 08:00:20 CDT 2011
On 08/26/2011 12:45 PM, Joerg-Cyril.Hoehle at t-systems.com wrote:
> Scott Ritchie wrote:
>> There was a bit of a philosophical discussion on #winehackers about the
>> merits of creating tests for functions that might be testing undefined
>> or unimportant behavior. Windows behaves one way, we behave another,
>> the tests measure this delta, but it's unknown if this will actually
>> improve a real world application.
> Vincent Povirk wrote an excellent reply. I hope somebody will turn
> that into a Wiki page.
>> More broadly, should we resist any change without a particular
>> (real-world) application in hand that needs it?
> This is stupid because you're always late. Like a fireman.
> a) Somebody who wrote the bug report is waiting for the bug to be fixed ASAP.
> b) It takes a lot of effort to analyse huge and cryptic log files to
> find out what the particular delta is.
>> Or should we err on the side of testable behavior,
> What you test is what you know. It's not erring.
> If you don't know what the behaviour should be, you can't add a
> FIXME pointing out that Wine deviates from native behaviour.
> When I asked about testing depth w.r.t. the MCI a few years ago, Paul
> Vriens advised to test to the bones. IMHO that's good advice.
It is a good advice. But that doesn't means one has to submit tests that
don't make sense.
> I recommend beginners to start writing tests *first*. Doing so, you
> learn a lot about the component Wine purports to mimic.
> You can then even start to predict failures!
> Somehow I believe I'm a very untypical Wine contributor. I'm not bug
I'm not bug driven either,
> driven. Quite to the contrary, my thesis is that once you've tested a
> component to the bones *and* replicated the observable behaviour,
but here I heartily disagree. You don't test nor replicate all the
observable behavior; that is way way too broad. An extreme case would be
to try to test how much time/CPU cycles a function call takes and trying
to replicate that in Wine. In general that doesn't matter but is is
observable and testable.
What you do want to test is:
- the de jure API (as documented), and
- the de facto API (as used by applications and thus kept "broken"
across Windows version).
So if an application relies on the fact that a function call will crash
with an invalid combination of arguments than Wine will have to crash
too but it doesn't matter if the segfault happens at address 0x00000002
> there will be few opportunities left for people to report bugs.
You sub-estimate the ingenuity of application developers to misuse the
API. Or you found a way to solve the halting problem ;)
> The basic alternative is that if you have free time now, you can
> either try to understand obscure logs of some component you're not
> exactly familiar with and more or less well documented, perhaps
> single-step your way through the debugger, *OR* you can spend time
> shedding light on said component via tests, mostly working with logs
> you control because you're working with full source to your test and
> the Wine code.
It is not an either or but a "do both", for the project in general that
is. People are free to scratch their itch as long as they don't make the
code a mess.
More information about the wine-devel