Introducing WineTestBot

Greg Geldorp ggeldorp at vmware.com
Mon Nov 30 03:05:45 CST 2009


Or, for those who like to be buzzword compliant, the Wine Testing Cloud
Service.

Inspired by all the talk about testing during WineConf 2009, I set out to fill
a gap in Wine testing. Most of the Wine developers don't have access to a
battery of Windows machines to run the tests they write/modify on. As a result,
there are quite a few regressions in the tests themselves. The WineTestBot
provides a whole bunch of Windows machines, ready to accept and run your
modified tests. In its current state you can submit Windows executables (most
likely cross-compiled on a Linux box) to the bot, which will run it in VMs of
your choice.

So, how does this work in practice? First, go to
https://winetestbot.geldorp.nl and click on "Register" to register an account.
Once your account is approved (you'll receive a notification via email) you
can log in. After log in, the menu in the sidebar changes to include an item
"Submit job". This allows you to upload an .exe from your local system to the
bot. You can pick the VMs (Windows versions) you want to run the test on. From
the "Home" page you can see the status of your job, by clicking on the job id
you get a bit more info about the progress. Once the test is completed, the
stdout generated by the test is retrieved from each VM and made available via
that web page.

The tests are run on VMware ESX VMs, so it's probably not all that useful to
run sound and DirectDraw tests on them. Also, no Win9x VMs are available.
I haven't set up 64-bit VMs yet, but those should be easy to add. Before each
test run, the VMs are reset to a clean snapshot, so no cruft from old tests
will be left around. Your test shouldn't take more than 5 minutes, which is
more than the WineTest framework allows anyway.

For each Windows version, there is a "base" VM and there can be additional
VMs too. The idea is that the base VM is running the most up-to-date and
most often used variation of that Windows version, so running your test on
the base VMs should give you a good indication on how the test behaves.
Additionally, you can run your tests on the other variations too. So, for
example for XP the base VM is XP Professional SP3, but there are additional
VMs like SP2, Media Center Edition, and a non-English version available too.
Obviously, even when you use all available VMs you'll cover a very small
subset of all possible configurations (most notably you won't test on a
single physical system) so please do check http://test.winehq.org after your
patch has been accepted to see if you introduced failures on other machines.

The plan for the future is to extend this a bit. Speaking from my own
experience, I know that programmers like to be as lazy as possible. Which
means that having to cross-compile your test before you can send it off to the
bot will be seen as a bit of a hurdle. So the next step will be to allow you
to upload the git diff instead of a Windows executable. The bot will apply
your diff to a git tree and take care of creating the cross-compiled Windows
executable, which it will then run on the Windows VMs. Step 3 will be to have
the bot watch wine-patches for anything that might impact the tests and do the
apply diff/cross-compile/run thing automatically. Obviously I'll be looking at
Dan's patch watcher for this 3rd step.

Security is a concern, but perhaps more for me than for you :-). I have
to trust you not to abuse this system. If you want to do something naughty
please do it on your own machine. There are some safeguards in the system,
like the timeout. In general, you should not expect any privacy, I do have
access to the consoles of the VMs and firewall logs. If you want to use this
system for something else than Wine tests, please talk to me first.

I'd like to thank Paul Vriens who acted as beta tester and made some valuable
suggestions. I welcome feedback so if you have any ideas on how to improve
this thing further please let me know.

Ge.




More information about the wine-devel mailing list