GSoC '10 Project Proposal: Buildbot Automated Testing System

[Arjun Comar]

Hey all, I wanted to run my proposal for GSoC by everybody for some feedback
and commentary, especially since student apps start in just 10 days. Any
advice is greatly appreciated.

GSoC 2010 Proposal: Buildbot Automated Testing System
For those who aren't aware, http://buildbot.net/ provides an automated
testing suite. I propose to set up a Buildbot system for Wine, using both a
periodic scheduler, and a try scheduler. I also intend to create an easy
configuration scheme that should keep maintenance fairly simple. In addition
the test suite should be modular and easily extensible, ideally building on
AppInstall and the existing test suite, as well as providing a means for new
tests to be hooked in.

--Buildbot--
To keep from seeming pedantic, I'll be brief with this one. Automating
testing suites are awesome and help keep down the number of bugs/regressions
that pop up. It'd be really cool if Wine had something like this.

--Schedulers--
The periodic scheduler's purpose is to run the full test suite at particular
time intervals. The timing on this one would need to be decided by the
project. For example, it could be set up to run the full test suite half a
week or so prior to the bi-weekly releases, or perhaps once a night if
people care to read the logs that often.

The try scheduler serves a slightly different purpose, namely to test on a
patch by patch basis. Initially, this means that the patch would have to be
directly uploaded to the test machine, though perhaps integration with
patchwatcher may be possible (though I'm told that patchwatcher is fragile
at best). Naturally, the full gamut of tests is probably not required for
every single patch, and so a more selective (and fast) subset of the test
suite can be run on the codebase after each patch, perhaps as limited as a
compilation check.

--Configuration--
Currently, one of the downsides of buildbot is that setup and configuration
is all done by editing a python script that initializes all the various
variables. This is clunky and adds a dependency to the person maintaining
the system. Since the configuration script is full python, configuration can
be moved out to a simple sh file (like most other Linux programs), and
parsing done in the current configuration script.

--Test Suite--
I would like to break apart the test suite into several, modular, components
that can then be strung together in some order for a particular test run.
Naturally, I intend to take advantage of the current test suite, and make
use of AppInstall, integrating the two together to provide a comprehensive
test suite. The conformance tests are also something that could potentially
be included. Tying into the previous section, I'd also like to make it easy
to attach new tests / test suites as people come up with new and better ways
to stress test various things. This is the portion that may wind up being
too much for a single person to do over the summer, though I doubt it.
Buildbot appears to have decent support for this kind of thing, so it may
just be wrestling it into doing what we need.

Also, I should have a sample buildbot system ready to showcase soon. I'm
currently testing it on a machine other than mine, but due to time
constraints this weekend, I probably won't be able to send it out until
early next week.


Thanks for taking the time to read this. Once again, any advice is greatly
appreciated. If you'd like any more detail or information, feel free to ask.
It's entirely possible I forgot to include something. I look forward to
working with all of you in the months to come.

-- 
Arjun Comar, Rose-Hulman '12
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20100319/00fd198a/attachment.htm>