Introducing WineTestBot

[Austin English]

On Mon, Nov 30, 2009 at 3:21 AM, Paul Vriens <paul.vriens.wine at gmail.com> wrote:
> On 11/30/2009 10:05 AM, Greg Geldorp wrote:
>>
>> Or, for those who like to be buzzword compliant, the Wine Testing Cloud
>> Service.
>>
>> Inspired by all the talk about testing during WineConf 2009, I set out to
>> fill
>> a gap in Wine testing. Most of the Wine developers don't have access to a
>> battery of Windows machines to run the tests they write/modify on. As a
>> result,
>> there are quite a few regressions in the tests themselves. The WineTestBot
>> provides a whole bunch of Windows machines, ready to accept and run your
>> modified tests. In its current state you can submit Windows executables
>> (most
>> likely cross-compiled on a Linux box) to the bot, which will run it in VMs
>> of
>> your choice.
>>
>> So, how does this work in practice? First, go to
>> https://winetestbot.geldorp.nl and click on "Register" to register an
>> account.
>> Once your account is approved (you'll receive a notification via email)
>> you
>> can log in. After log in, the menu in the sidebar changes to include an
>> item
>> "Submit job". This allows you to upload an .exe from your local system to
>> the
>> bot. You can pick the VMs (Windows versions) you want to run the test on.
>> From
>> the "Home" page you can see the status of your job, by clicking on the job
>> id
>> you get a bit more info about the progress. Once the test is completed,
>> the
>> stdout generated by the test is retrieved from each VM and made available
>> via
>> that web page.
>>
>> The tests are run on VMware ESX VMs, so it's probably not all that useful
>> to
>> run sound and DirectDraw tests on them. Also, no Win9x VMs are available.
>> I haven't set up 64-bit VMs yet, but those should be easy to add. Before
>> each
>> test run, the VMs are reset to a clean snapshot, so no cruft from old
>> tests
>> will be left around. Your test shouldn't take more than 5 minutes, which
>> is
>> more than the WineTest framework allows anyway.
>>
>> For each Windows version, there is a "base" VM and there can be additional
>> VMs too. The idea is that the base VM is running the most up-to-date and
>> most often used variation of that Windows version, so running your test on
>> the base VMs should give you a good indication on how the test behaves.
>> Additionally, you can run your tests on the other variations too. So, for
>> example for XP the base VM is XP Professional SP3, but there are
>> additional
>> VMs like SP2, Media Center Edition, and a non-English version available
>> too.
>> Obviously, even when you use all available VMs you'll cover a very small
>> subset of all possible configurations (most notably you won't test on a
>> single physical system) so please do check http://test.winehq.org after
>> your
>> patch has been accepted to see if you introduced failures on other
>> machines.
>>
>> The plan for the future is to extend this a bit. Speaking from my own
>> experience, I know that programmers like to be as lazy as possible. Which
>> means that having to cross-compile your test before you can send it off to
>> the
>> bot will be seen as a bit of a hurdle. So the next step will be to allow
>> you
>> to upload the git diff instead of a Windows executable. The bot will apply
>> your diff to a git tree and take care of creating the cross-compiled
>> Windows
>> executable, which it will then run on the Windows VMs. Step 3 will be to
>> have
>> the bot watch wine-patches for anything that might impact the tests and do
>> the
>> apply diff/cross-compile/run thing automatically. Obviously I'll be
>> looking at
>> Dan's patch watcher for this 3rd step.
>>
>> Security is a concern, but perhaps more for me than for you :-). I have
>> to trust you not to abuse this system. If you want to do something naughty
>> please do it on your own machine. There are some safeguards in the system,
>> like the timeout. In general, you should not expect any privacy, I do have
>> access to the consoles of the VMs and firewall logs. If you want to use
>> this
>> system for something else than Wine tests, please talk to me first.
>>
>> I'd like to thank Paul Vriens who acted as beta tester and made some
>> valuable
>> suggestions. I welcome feedback so if you have any ideas on how to improve
>> this thing further please let me know.
>>
>> Ge.
>>
>>
>>
>
> Let me be the first to thank Ge for this awesome piece of work.

Seconded! On a slightly related note, would it be possible to add a
16-bit display VM? I finally got around to testing wine on a 16-bit
display again, and several things have regressed. It would be good to
test the testsuite on windows in that configuration to see what
happens.

-- 
-Austin