WineTestBot test coverage

[Francois Gouget]

When setting up the WineTestBot VMs I have tried to make it so we 
can compare their results across Windows versions so we know if a 
specific behavior is specific to a given Windows version or was 
introduced in a specific Windows version. At the same time I tried to 
diversify the Windows configurations to get better coverage of 
various locales, particularly when they involve right-to-left text or 
asian characters, but also to test other aspects like having multiple 
network cards, two disks, etc.

While ideally the WineTestBot would be able to switch each VM between 
multiple configurations (through the use of snapshots), currently this 
is not supported. Thus we can only test as many configurations as we 
have VMs (and Windows licenses). As a result choices have to be made, 
the coverage is not complete and there may be room for improvement.

For instance w7u is configured for Japanese because it's an asian 
language and I know we have developers who know Japanese. But it turns 
out a number of test failures are specific to Korean. Ideally we'd have 
both but there are only two VMs(*) that can be configured for 
non-English locales and the other one is configured to check for 
right-to-left issues (previously Arabic and now through Hebrew).

Similarly all VMs run the tests with elevated privileges because that's 
what allows running the maximum number of tests. But this means tests 
are never tested in unprivileged accounts leading to a bunch of failures 
when that happens.

So should we change the configuration of some of the WineTestBot VMs? 
Which aspects are the most important?

An additional note is that I can relatively easily reconfigure a VM and 
then switch between that configuration and what we have now. So for 
instance I could switch the Japanese VM to Korean if a developer needs 
it to debug things, and then switch it back to Japanese when he's done. 
Between the timezone delays and WineTestBot's own, the latency would be 
around a day.


So here is a summary of the available configurations. Let me know if 
there are changes to do:

w2000pro  IE 6 sp1, DirectX 9.0c, MSXML 4&6, VC 2005/8/10, IPX, AppleTalk, data CD
wxppro    IE 8, DirectX extras, MSXML 4, VC 2005/8/10, winpcap
w2003std  IE 8, .Net 2.0 SP2, 3.0 SP2, 3.5 SP2, 4, MSXML 4, VC 2005/8/10.
wvista    IE 9, .Net 3.5 SP1 & 4.5.1, RMP
wvistau64 IE 8, Hebrew, tests on e:, DirectX extras, MSXML 4, VC 2005/10. winpcap 
w2008s64  IE 9, .Net 4.0, no sound, two network cards
w7u       IE 11, Japanese, .Net 3.5 & 4.5, VC 2012/13, data CD
w7pro64   IE 11, .Net 4.5.1
w8        Win 8.1, IE 11, DirectX extras, MSXML 4, VC 2005/10.
w864      Win 8.1, IE 11, DirectX extras, MSXML 4, VC 2005/10.
w1064     Edge, DirectX extras

Legend:
 - DirectX extras: This installs some dlls that are not installed by 
       default, for DirectMusic and DirectPlay for instance.
 - AppleTalk: The AppleTalk network protocol.
 - IPX: The IPX network protocol.
 - RMP: The Reliable Multicast Protocol network protocol.


Known holes:
 * No Windows 8 (only Windows 8.1).
 * No IE 7 configuration.
 * No IE 10 configuration.
 * No non-elevated or regular user tests.
 * Korean locale. Other locales?
 * No mixed locale (to ensure the tests check the right locale API).


(*) Though the locale of the new Windows 10 VM is probably configurable 
    too. So that would be 3 VMs. But do we want to add failures to a VM 
    that already has a lot?


-- 
Francois Gouget <fgouget at codeweavers.com>