Wine Test analysis, report

[Francois Gouget]

On Fri, 25 Apr 2014, Jeremy White wrote:
[...]
> I'm waiting for Francois to report that things are essentially operational
> again; until I get that word, I'm not planning to beat the drum.

Here's the belated status update.
In short the WineTestBot is essentially operational.

 * I've put the Vista 64-bit VM back online yesterday. Since it's an 
   Ultimate edition I configured it for Arabic so we have a RTL test 
   configuration. That VM also runs the tests from the E: drive instead 
   of the usual C: drive so if tests make incorrect drive assumptions we 
   should catch them. Finally it also has WinPcap and the various extar 
   DirectX, msxml and Visual C++ runtime components.

 * I have not checked the revert times yet. They are definitely not 
   catastrophic but I'm not sure they are very good either (maybe
   on the order of 1 to 3 minutes for some VMs but I could be wrong).

 * There are some VM configuration changes that could be useful. For 
   instance it seems testing Chinese rather than Japanese would better 
   exercise the font association code. Also it would be interesting to 
   have the systray in a non default position. Working those in should 
   not be too hard so I'll try to do it at some point.

Concerning the test results:

 * The MSI tests are timing out on XP, Windows 2008 and Windows 7 
   64-bit. I'm using the same disk configuration as before so they have 
   no particular reason to be slow. Since the timeouts don't happen with 
   all the VMs, maybe it's a Windows configuration issue rather than a 
   VM configuration one.

 * The ntdll:exception test crashes instead of having failures. That's 
   because we switched the processor form AMD to Intel. Huw submitted a 
   patch that should fix this so there's hope this will be fixed at some 
   point.

 * The results of the 64-bit Windows 8 VM are being ignored. That's 
   because it does not complete the tests within the alloted 30 minutes.
   The fault with dpnet:address, dpnet:peer, dpnet:server, msi:action, 
   msi:msi which all time out, accounting for 10 minutes. I forgot to 
   retweak the Windows 8 configuration to avoid the dpnet timeouts. I 
   guess I'll have to readd them.

 * The 32-bit Windows 8 VM seems to complete the tests in time but it 
   has 52 failures for the one run I checked and thus its results are 
   ignored too.

But for now I will focus on fixing the issues with the WineTestBot 
engine code: if I ask Alexandre to restart it again he'll blow a gasket 
;-) Also I want to put an end to the network read timeout errors so we 
no longer get boterrors. And when an error happens I also want the 
WineTestBot engine to better handle them, for instance if there was a 
network outage while running a test, that test should be requeued for 
when the network is back.

-- 
Francois Gouget <fgouget at codeweavers.com>