Fixing crashes in Tests (OS version check)

[Rolf Kalbermatter]

Juan Lang wrote:

> So, in my opinion, the general principle is this:  
> if the behavior is very different on different Windows 
> versions, and if the behavior on some Windows version is to 
> crash, it seems reasonable to remove the test, as: 1) we want 
> our tests to reflect Windows behavior that applications count 
> on, and different behavior (especially crashing) often 
> implies the applications don't count on a particular 
> behavior, and 2) having regression tests crash is 
> undesirable, as many other valid tests will not be run on a platform.

I don't think you can fully argument with this nowadays. There are
many applications out there that specifically state to not support
pre W2K or sometimes even XP and often refuse to even install on
older systems. So in this case they could be perfectly happy with
passing a NULL value to that function and since they never have
been tested on an older system the developer wouldn't even know
that there exists a potential problem.

So I would think it's useful to test that the Wine API does not
crash on a NULL parameter since newer Windows versions won't do
that either. 

> This doesn't mean that we can't make Wine not crash given the 
> bogus input, it just means we may not be able to have a 
> regression test for this behavior.  And, of course, there may 
> be specific instances in which we wish to test potentially 
> crashy behavior, but I'd argue that they better be really 
> important to override points 1) and 2).

This case seems fairly simple to me. As long as Wine strives to
provide support for Win9x and NT4 it needs to make efforts to
support both the newer and the older behaviour. For the API
itself this means allowing NULL values to be passed without
crashing and for the tests it means skipping that particular
test for those versions that do crash on it. And while a version
test seems the easiest I can also understand why the tests
should not have such artificial dependencies especially since
these tests would have to distinguish between running on
Windows or Wine set to emulate Win9x. 

But here it seems fairly easy to work around this problem. The
fact that certain Snmp APIs are not implemented or possibly return
an ERROR_UNIMPLEMENTED status on those platforms should be enough
to determine if this test should be skipped.

>And coming back to this test:  I still think it's easier just
>to remove the test than to come up with a hacky way of avoiding
>it on a crashy system.

No argument with the being easier part. But not writing tests at
all would be also easier, but I do not think that is a valid argument
to not do them. Typically to do a good test costs more time than an
actual implementation of a function and Wine development could
be sped up by not writing tests. But it would only implement more
functionality with a lot more errors (or should that be less bugs :-).

Rolf Kalbermatter