Regression testing breakthrough

[Damjan Jovanovic]

Hi

Since the beginning, I've had issues with regression testing. Despite the
fact it's very useful, it takes forever, it's easy to make a mistake
(especially during "reverse regression testing"), users find it too long and
technical, and only a small minority of regressions are ever bisected. And
several patches need backporting to allow older versions of Wine to compile
and run on today's make, gcc, and libraries - this is the case even for the
1.0.x releases from less than 3 years ago!

The problem is of course compilation. "configure" takes at least 40 seconds,
without any way to speed it up on multi-core CPUs. "make" takes > 5 minutes,
and it's only taking longer as Wine gets bigger. Compilation is
fundamentally complex and technical to users.

But what if we had precompiled binaries, and regression testing consisted of
just running different versions of Wine?

Wine binaries take up about 122 MB and take over 5 minutes to compile.
There's now 35770 commits between 36def4af0ca85a1d0e66b5207056775bcb3b09ff
(Release 1.0) and "origin". That's about 4.4 terrabytes of storage and over
4 months of compilation, if each of those versions had to be compiled and
installed into its own prefix, way beyond what most users are willing or
able to store or do. Most patches however end up affecting only a few binary
files in the end, and compiling successive versions allows "make" to be very
quick.

So I've written a tool that compiles Wine and adds each commit's binaries
into a Git repository. It knows how to compile old versions of Wine
(currently as far back as 1.0). It knows that commits affecting only
ANNOUNCE, .gitignore, and files in dll/ or programs/ ending with .c and such
don't need to go through the endlessly slow "configure", only "make". It is
stateless: if interrupted, it can resume from the last successful commit. It
works around bugs in GNU make (you won't believe how many there are...).

This tool compiled all 35000 or so commits from Wine 1.0 to around 4th
October 2011 in only 7 days, generating a Git repository of Wine binaries
that's only 26 gigabytes in size. Regression testing with binaries is a
pleasure: it takes only a few seconds :-) on each bisection. I bisected a 16
step regression in just 20 minutes, and most of that time was spent running
the application and dealing with 2 X-server crashes.

I haven't figured out how to make the binaries available to users. Few users
can clone a 26 gigabyte repository, and even fewer places can serve that
much to multiple users. Maybe Git can compress it further? The other idea I
had is that users should be able to regression test through a GUI tool.
Maybe the GUI tool can just download and run the +/- 122 MB binary snapshots
for specific commits, instead of having the entire binary repository
locally?

Any other ideas? Would you like to see this tool? Can I send an attachment
with it?

Thank you
Damjan Jovanovic
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20111018/21a6c166/attachment-0001.html>