Extending the testframe to cope with different windows versions

[James Hawkins]

On 11/25/06, Paul Vriens <paul.vriens.wine at gmail.com> wrote:
> On Fri, 2006-11-24 at 19:06 +0100, Kai Blin wrote:
> > On Friday 24 November 2006 13:33, Paul Vriens wrote:
> > > Hi,
> > >
> > > after my last patch to yet 'remove' another test I had another look at
> > > tests.winehq.org. With the arrival of IE7 and of course a lot of changes
> > > to dll's (not even mentioning Vista) it looks like Windows versions are
> > > starting to behave not even close to similar.
> >
> > Do we really need to test all windows versions? Looking at the test page,
> > there seems to be no real consistent pattern of windows boxes running the
> > tests.
>
> Testing on all windows versions will at least give us an idea how dll's
> are implemented there. We do have to find a common denominator if
> possible. We will thus have no version specific implementation unless
> proven that it's needed.
> So maybe in the end it's not so much the versions of windows but rather
> the versions of dll's.
>
> >
> > > How can we deal with these differences? Several tests we have already
> > > include checks against a lot of possible return values or errorcodes.
> > > It's a shame when we have to stop running a test because it fails on 1
> > > or more versions of Windows.
> >
> > Well, usually something like this means that applications will fail if they
> > can't deal with this. I'm not sure if it's too much of a problem.
> >
>
> The point I was trying to raise is when we exclude the test we cannot
> guarantee we are doing things correctly. It could be that a newer dll
> doesn't check for NULL anymore. It could also be that a dll exhibits a
> bug. It's the latter I'm 'worried' about as nobody will try to
> incorporate that test anymore.
>
> > Cheers,
> > Kai
> >
>
> Maybe instead of a '#ifdef 0' or 'if (0)' we should try to make the
> exclusion/inclusion of tests better.
>
> For example: I've found that the advpack dll that comes with IE7 behaves
> different then before. Instead of doing the 'if (0)' that's done now we
> could only test on a dll version of advpack lower then a certain
> revision (in this case 7.0.5730.11).
>
> There's still no need to put these kinds of things in the Wine code
> until proven that it's needed. I'm only talking about the conformance
> tests.
>

The if 0 exclusion is enough because it shows that at one point
advpack checked for the null parameter, but now it doesn't on some
versions.  We check for the null param, so we're on the safe side of
the fence.  Not many apps/installers depend on a certain function
crashing with a null param, though there are one or two we've seen.
advpack is almost completely undocumented and is only used by the IE
and mdac installers (basically), so this issue is inconsequential.
Also, Alexandre has specifically said that he doesn't want the tests
filled with version checks.

-- 
James Hawkins