Fixing crashes in Tests (OS version check)

[Reece Dunn]

2008/5/31 Jeff Latimer <lats at yless4u.com.au>:
> James Hawkins wrote:
>> On Fri, May 30, 2008 at 11:18 PM, Alistair Leslie-Hughes
>> <leslie_alistair at hotmail.com> wrote:
>>
>>> Hi,
>>>
>>> I've being investigating crashes that occur on Win98 when the tests run.  I
>>> have fixed a couple already which where trivial. When I attempted to fix the
>>> snmpapi tests, I have found that i will end up removing alot of tests (~
>>> 17). If I add a check for the windows version, I can skip the tests that
>>> will crash, also there is already tests with if(0) around them. What is the
>>> policy about checking the version of the OS before running certain tests?
>> Version checks are not allowed.
> Is this actually a sensible policy?  As Windows version age (Win95
> arrived 12 years ago in 96!) there are going to be more and more
> incompatibilities with later versions.  Windows is not backward
> compatible.  Apps built for Vista are not guaranteed to run on earlier
> versions and the Windows api moves along.  The upshot is that Wine tests
> must be dumbed down (ie. we accept multiple results (Win95 and above) or
> we delete the test).  The outcome is that we don't actually know what we
> are testing as Wine could be producing results for XP, Vista, W2K but it
> tests ok.

The reason applications built for Vista are not guaranteed to work on
earlier versions is that they depend on new API available in Vista.
For example, they may use the new 'glass' API directly, rely on the
new menu button support in the button control, or some new behaviour
in an existing API. The only case that may cause issues here in the
tests is the last one, as Wine may be validly conforming to XP or
earlier behaviour.

Note though that most of the differences are in what error codes are
returned. Applications in Windows should not depend on specific error
codes being returned (as they may vary from version to version as the
Wine tests verify), just whether the call succeeded or failed, unless
those error codes are specifically called out in the documentation -
QueryInterface returning E_NOINTERFACE if an interface is not
supported by the COM object, for instance.

Also with the error codes, they may change with each hot fix, service
pack and application (e.g. Internet Explorer) update. For example, the
shlwapi istream tests have one case that returns E_NOTIMPL on Windows
2000 except when IE6 or later is installed (i.e. the shlwapi version
has been updated to match the one on XP or later).

Older versions of Windows don't expose certain API. In this case (as
is such in this instance), the tests should not crash but should be
skipped with a message saying that the function(s) are not available.
This scales better, since the API may not be available on a range of
Windows versions, or the API may be added when installing IE6 or IE7.
New window and user/common control messages can be handled in a
similar way, although I don't know the exact mechanics of how this
would work.

The interesting case is were you have specific behavioural differences
(e.g. crash when passed a NULL argument vs returning an error code
such as E_POINTER). In these cases, Wine is free to choose what it
does as Windows is not consistent with itself (see the application
compatibility issues with Vista). Here, if a specific application is
dependant on certain behaviour, Wine should support that and have that
documented in the tests.

Writing tests that work on all Windows platforms is hard. That does
not mean that the tests should use OS checks and I think the existing
methodology is the right approach to writing the tests. There will be
cases where things can't be tested due to incompatible differences,
and that would be an area that could be investigated to see if those
can be improved (such as the new buggy functionality and possibly
others in that family) to allow support for older documented behaviour
in the tests, while keeping the Wine implementation sane.

- Reece