Extending the testframe to cope with different windows versions

[Paul Vriens]

On Fri, 2006-11-24 at 19:06 +0100, Kai Blin wrote:
> On Friday 24 November 2006 13:33, Paul Vriens wrote:
> > Hi,
> >
> > after my last patch to yet 'remove' another test I had another look at
> > tests.winehq.org. With the arrival of IE7 and of course a lot of changes
> > to dll's (not even mentioning Vista) it looks like Windows versions are
> > starting to behave not even close to similar.
> 
> Do we really need to test all windows versions? Looking at the test page, 
> there seems to be no real consistent pattern of windows boxes running the 
> tests.

Testing on all windows versions will at least give us an idea how dll's
are implemented there. We do have to find a common denominator if
possible. We will thus have no version specific implementation unless
proven that it's needed.
So maybe in the end it's not so much the versions of windows but rather
the versions of dll's.

> 
> > How can we deal with these differences? Several tests we have already
> > include checks against a lot of possible return values or errorcodes.
> > It's a shame when we have to stop running a test because it fails on 1
> > or more versions of Windows.
> 
> Well, usually something like this means that applications will fail if they 
> can't deal with this. I'm not sure if it's too much of a problem.
> 

The point I was trying to raise is when we exclude the test we cannot
guarantee we are doing things correctly. It could be that a newer dll
doesn't check for NULL anymore. It could also be that a dll exhibits a
bug. It's the latter I'm 'worried' about as nobody will try to
incorporate that test anymore.

> Cheers,
> Kai
> 

Maybe instead of a '#ifdef 0' or 'if (0)' we should try to make the
exclusion/inclusion of tests better.

For example: I've found that the advpack dll that comes with IE7 behaves
different then before. Instead of doing the 'if (0)' that's done now we
could only test on a dll version of advpack lower then a certain
revision (in this case 7.0.5730.11).

There's still no need to put these kinds of things in the Wine code
until proven that it's needed. I'm only talking about the conformance
tests.

Cheers,

Paul.