Towards Zero Test Suite Failures

[Francois Gouget]

On Mon, 14 Jan 2008, Stefan Dösinger wrote:

> Am Montag, 14. Januar 2008 14:41:35 schrieb Reece Dunn:
> > But if you fix a test failure in Wine that is failing on Windows, then
> > you are introducing a bug in Wine.
> Not necessarily. Some tests are too strict in what they expect, and sometimes 
> the Windows behavior is "wrong", in the way that some application expects 
> something different and wouldn't run on this Windows installation either.

But the only way to know is to check the Windows results. So while 
you're doing that you can as well try to fix the test on Windows.


[...]
> This Examples of "broken" Windows installations are VMware 
> installations with Direct3D support enabled. VMWare has a D3D driver 
> that works similarly to Wine, but has a few bugs that our tests 
> stumble uppon.

<rant>

This gets mentioned *every single time*.

For what it's worth, as far as I know none of the tests I run (the 
'fg-xxx' ones) has that D3D driver you speak of.

They still have failures in the Direct3D tests but if you check the 
Windows XP results, you will see that my logs show only two Direct3D 
tests failing. That's very little compared to the 73 other tests that 
fail... for the Windows XP results alone.

So I'd like people to focus a little bit less on these two Direct3D 
failures and a little bit more on the 73 other failures. That the tests 
are being run in VMware is no excuse for not looking at them. If these 
73 failing tests were fixed that would be a huge step for the 
conformance test suite.

</rant>

-- 
Francois Gouget <fgouget at free.fr>              http://fgouget.free.fr/
                 There are 10 types of people in the world...
               those who understand binary and those who don't.