WineTestBot status update

[Francois Gouget]

We switched to the new WineTestBot near the start of September. Early on 
it ran into a problem when it was flooded with tests due to the 
__WINESRC__ tests work and it failed to keep up.

The reason is that as time went on the time it takes to get the VMs back 
to a clean state ready to run the tests (the revert time) kept 
increasing as can be seen on this graph:

http://fgouget.free.fr/wtb/vm1-revtimes.png

There one can see the increased activity in september that what was 
caused by the surge in test patches. But the most remarkable feature is 
the Windows XP revert times which increase linearly, reaching over 30 
minutes. The revert times for the other VMs are much more random but 
still seem to increase somewhat (1).

The reason for the increase is still unknown. The revert time fell down 
a few times and now I suspect this corresponds to times when the VM was 
restored from backup.

So at the start of October I upgraded to QEMU 1.6.0 and restored all the 
VMs from backup, hoping this would solve the problem. The revert times 
did get much better and the WineTestBot now manages to keep up:

http://fgouget.free.fr/wtb/vm1-revtimes-new.png

The upgrade happened around the 2nd of october and the revert times fell 
at that point. But looking at more recent data it's clear that the 
Windows XP revert times are increasing again, while those of the other 
VMs remain stable (see vm1-revtimes.xls for the raw data).

That XP VM is one of the first ones I created, so it probably was 
created with a pretty old QEMU version. Futhermore it was only an out of 
the box SP2 and did not have any of the later updates. So I decided to 
recreate it from scratch, using QEMU 1.6.0, and to add all the other 
updates as I was at it. And I'm hoping that this time things will be 
better. That's why this VM is currently in 'maintenance' mode.

I also took down the 32-bit Windows 8 VM for maintenance. It too is 
missing proper updates and the plan is to upgrade it to Windows 8.1.


Another issue has been that the tests are running a bit slow, a 
particular point of pain was msi:action and msi:install which were 
regularly timing out. See:

http://fgouget.free.fr/wtb/runtimes.htm

So while rebuilding the Windows XP VM I did some performance tests. That 
paid off. I found that switching the emulated disk cache mode from 
writeback to default roughly halves the msi run times:

http://fgouget.free.fr/wtb/runtimes2.htm

msi:action and msi:install went from an average of 99 and 90 seconds 
down to 48 and 39 seconds and they are not timing out anymore. The 
remaining troublesome tests are now ddraw:ddraw{2,4,7}, shell32:shlexec, 
and winspool.drv:info.

I also uncovered two other potential performance issues:
 * Some VMs have Windows Defender and apparently it was starting a full 
   disk scan as soon as the VM was ready to run the tests. That's 
   clearly not good so I disabled Windows Defender in all VMs.

 * Most VMs were still using 20% of the CPU when idle. That's fine if 
   only one VM is running but could impact performance (through 
   increased interrupt rates) if many are running at once (like on the 
   WineTestBot). Apparently the culprit is the USB Tablet device that's 
   added by default. So I removed it from all VMs too (and now the mouse 
   is even more laggy than before when I work on the VMs :-( But that 
   does not impact the tests).

So there is still a lot of work to do on the WineTestBot, and that's 
not even counting fixing all the failing tests.


(1) It's almost as if the increasing Windows XP revert times were 
pulling up the other VM revert times. Yet there's only one revert at a 
given time, so I don't see how this would be possible.

(2) Setting the cache mode to writeback was improving the performance 
quite a bit with the pre-upgrade QEMU version. But obviously it's not 
true anymore. Another interesting point is that the Virtio mode improves 
performance by a factor of 10 (down from 450 seconds to 45 in 
msi:action). Yet Windows 8 posts the fastest times yet in the basic IDE 
mode (~23 seconds). I couldn't get Windows 8 working with the Virtio 
disk yet and I have not verified that it's really doing the same set of 
tests. So that remains to be confirmed.

Here is some performance data taken from the Windows XP VM while the 
TestBot was idle. From it cache=default looks like it's a completely 
separate mode!

msi:action | msi:install | msi:msi
448 460 520| 487 439     | 238 229  VGA, qcow2 ide disk, cache:default
44 45 42   | 47 49 46    | 24 23 21 VGA, qcow2 virtio disk, cache:default
48 45 50   | 47 48 48    | 23 23 24 VMVGA/VGA, qcow2 virtio disk, cache:default
47 46      | 48 46       | 23 22 22 VMVGA, qcow2 virtio disk, cache:default
47 47      | 49 48       | 24 23    VMVGA, raw virtio disk, cache:default
62 67 67   | 64 60 63    | 29 27 27 VMVGA, raw virtio disk, cache:none
111 92 72 113 | 101 101  | 37 37 40 VMVGA, raw virtio disk, cache:writethrough
61 64 53   | 70 59 58    | 36 28 29 VMVGA, raw virtio disk, cache:writeback

46 44      |             |  VMVGA, raw virtio disk, cache:default
71 55 68   |             |  VMVGA, raw virtio disk, cache:writeback
65 71 72 69|             |  VMVGA, raw virtio disk, cache:none
76 118 111 |             |  VMVGA, raw virtio disk, cache:writethrough
50 48 47   |             |  VMVGA, raw virtio disk, cache:default

And the preliminary Windows 8 performance. I hope it holds up.
msi:action | msi:install | msi:msi
23 22 22   | 16 15 16    | 13 13 15 VGA, qcow2 ide disk, cache:default


-- 
Francois Gouget <fgouget at codeweavers.com>