Fixing the conformance tests

[James Hawkins]

On Sat, Sep 6, 2008 at 5:19 AM, Paul Vriens <paul.vriens.wine at gmail.com> wrote:
> Hi,
>
> As we are slowly progressing towards a more green test.winehq.org there are
> still some conformance tests that fail either on all platforms or with a
> specific configuration (like IE7).
>
> This email is merely to list those and see if the SME's can have a look in these
> tests. (I didn't put to much effort in stuff below XP and left out user32)
>
> Tests with timeouts
> ===================
> - dplayx:dplayx fails with a timeout on all windows servers

According to Ismael, he has a few fixes that cut the time needed in
half.  Ismael, what's the status on this?

http://winehq.org/pipermail/wine-devel/2008-August/068125.html

> - msi:install fails with a timeout on a lot of windows servers
>

Working on it, but it makes me seriously grumpy.

> Tests that fail with IE7 installed
> ==================================
> - hlink:hlink has a consistent 8 failures on servers with IE7

Mike M.'s gone, so someone is going to have to step up to the plate on
this one.  It won't be that hard.

> - mshtml:htmldoc has 8 failures on servers with IE7 (and more, but also on others)
> - shdocvw:webbrowser has 3 failures on servers with IE7.
>

Jacek has stated he doesn't have much time or interest in fixing
these, so the plate calls again.

> Tests that fail starting with a specific OS (or level)
> ======================================================
> - msi:package has 16 failures on XP-SP3+ and 16 extra on W2K3-SP2+
> - ole32:storage32 has 24 failures on XP-SP3+ and 24 extra on W2K3-SP2+
> (both the above have to do with access rights on files)
>

Pretty easy.  I have a patch to send in tomorrow.

> Tests that fail on Windows and succeed on Wine
> ==============================================
> - ws2_32:sock has 1 consistent failure on all Windows boxes but not on Wine
>

Last attempt got no comments:

http://winehq.org/pipermail/wine-patches/2008-May/054547.html

> Tests with failures on all Windows boxes
> ========================================
> - riched20:editor has failures on all boxes

These are all pretty easy to fix.

> - urlmon:protocol either has timeouts or failures on all boxes

Jacek has said this is a known bug, but the fix is not easy.  Jacek,
can you elaborate on that?  Maybe someone else can take a look at it
if you're too busy.

> - urlmon:url fails with 6 consistent failures on all boxes
>

Just need someone to look at the results and write up a fix.

> The above is a subset of all the failures and are not the easy ones. There are
> still loads of tests that fail on only 1 OS or even on 1 server.
>

Besides the urlmon:protocol bug, the rest are all pretty easy to fix
(referring to the issues you brought up in this message).

Folks, fixing the tests is a relatively easy task.   It's also a great
way to become better acquainted with the code base and the development
process if you're new to working on Wine.  For current developers, if
you've written tests in the past, check out test.winehq.org every once
in a while and see if any of your tests are failing.  And yes, we do
care about failures in win9x.  The average number of test failures
today is 64.7, which is down from 72.4 a month ago*.  Let's get that
down to zero.

* Averages of each windows version were then averaged together.

-- 
James Hawkins