Introducing WineTestBot

[Paul Vriens]

On 11/30/2009 10:05 AM, Greg Geldorp wrote:
> Or, for those who like to be buzzword compliant, the Wine Testing Cloud
> Service.
>
> Inspired by all the talk about testing during WineConf 2009, I set out to fill
> a gap in Wine testing. Most of the Wine developers don't have access to a
> battery of Windows machines to run the tests they write/modify on. As a result,
> there are quite a few regressions in the tests themselves. The WineTestBot
> provides a whole bunch of Windows machines, ready to accept and run your
> modified tests. In its current state you can submit Windows executables (most
> likely cross-compiled on a Linux box) to the bot, which will run it in VMs of
> your choice.
>
> So, how does this work in practice? First, go to
> https://winetestbot.geldorp.nl and click on "Register" to register an account.
> Once your account is approved (you'll receive a notification via email) you
> can log in. After log in, the menu in the sidebar changes to include an item
> "Submit job". This allows you to upload an .exe from your local system to the
> bot. You can pick the VMs (Windows versions) you want to run the test on. From
> the "Home" page you can see the status of your job, by clicking on the job id
> you get a bit more info about the progress. Once the test is completed, the
> stdout generated by the test is retrieved from each VM and made available via
> that web page.
>
> The tests are run on VMware ESX VMs, so it's probably not all that useful to
> run sound and DirectDraw tests on them. Also, no Win9x VMs are available.
> I haven't set up 64-bit VMs yet, but those should be easy to add. Before each
> test run, the VMs are reset to a clean snapshot, so no cruft from old tests
> will be left around. Your test shouldn't take more than 5 minutes, which is
> more than the WineTest framework allows anyway.
>
> For each Windows version, there is a "base" VM and there can be additional
> VMs too. The idea is that the base VM is running the most up-to-date and
> most often used variation of that Windows version, so running your test on
> the base VMs should give you a good indication on how the test behaves.
> Additionally, you can run your tests on the other variations too. So, for
> example for XP the base VM is XP Professional SP3, but there are additional
> VMs like SP2, Media Center Edition, and a non-English version available too.
> Obviously, even when you use all available VMs you'll cover a very small
> subset of all possible configurations (most notably you won't test on a
> single physical system) so please do check http://test.winehq.org after your
> patch has been accepted to see if you introduced failures on other machines.
>
> The plan for the future is to extend this a bit. Speaking from my own
> experience, I know that programmers like to be as lazy as possible. Which
> means that having to cross-compile your test before you can send it off to the
> bot will be seen as a bit of a hurdle. So the next step will be to allow you
> to upload the git diff instead of a Windows executable. The bot will apply
> your diff to a git tree and take care of creating the cross-compiled Windows
> executable, which it will then run on the Windows VMs. Step 3 will be to have
> the bot watch wine-patches for anything that might impact the tests and do the
> apply diff/cross-compile/run thing automatically. Obviously I'll be looking at
> Dan's patch watcher for this 3rd step.
>
> Security is a concern, but perhaps more for me than for you :-). I have
> to trust you not to abuse this system. If you want to do something naughty
> please do it on your own machine. There are some safeguards in the system,
> like the timeout. In general, you should not expect any privacy, I do have
> access to the consoles of the VMs and firewall logs. If you want to use this
> system for something else than Wine tests, please talk to me first.
>
> I'd like to thank Paul Vriens who acted as beta tester and made some valuable
> suggestions. I welcome feedback so if you have any ideas on how to improve
> this thing further please let me know.
>
> Ge.
>
>
>

Let me be the first to thank Ge for this awesome piece of work.

-- 
Cheers,

Paul.