Towards Zero Test Suite Failures

[Reece Dunn]

On 14/01/2008, Dan Kegel <dank at kegel.com> wrote:
> On Jan 13, 2008 8:30 PM, Zachary Goldberg <zgold550 at gmail.com> wrote:
> > > On my machine, we've been hovering between
> > > five and ten test suite failures for some time
> > > (see http://bugs.winehq.org/show_bug.cgi?id=9916 )
> > > ...
> > > How 'bout folks spend some time tracking
> > > the current six odd failures down and cleaning them up?
> >
> > Sounds like a worthy goal IMO.  It seems theres much more than 5 or 6
> > things which need work though:
> > http://test.winehq.org/data/200801121000/
>
> Not a pretty picture, is it?
> But I was only referring to the tests that fail on Wine; there,
> we have control over both the test and the code, so if
> we can't get those tests passing, we're pretty weak :-)

But if you fix a test failure in Wine that is failing on Windows, then
you are introducing a bug in Wine.

> > Perhaps we should try and
> > have the nightly suite also pass all tests -- or is this too much of a
> > frivolous goal?
>
> Getting all tests passing on Windows XP would be great,
> but it's a lot of work.   It'd be less work if we focused on just the tests
> that are failing on *all* Win XP systems, but that's still
> at least 180 ok's that aren't ok.  Let's at least fix the 5 or 10 that are
> failing on Wine, and try to keep them fixed.

Ideally, all tests should pass on all platforms. However, there are
issues with different service packs/hot fixes on each version of
Windows, different user configuration (themes, UAC on/off, locale,
etc.), different hardware and even different behaviour between
different versions of Windows.

I like the idea of getting the tests passing on Windows XP (SP2?) -
that will then be the baseline for Wine. This should not cause
regressions on other versions of Windows.

Getting the tests passing on older verisons of Windows will be
important for backward compatibility (e.g. if you have a program that
only runs on Windows 3.1 or Windows 95), and will ensure that Wine
does not break in those cases.

- Reece