Regression testing breakthrough

[Ferenc Gergely Szilagyi]

On Tue, Oct 18, 2011 at 10:45 AM, Damjan Jovanovic <damjan.jov at gmail.com>wrote:

> Hi
>
> Since the beginning, I've had issues with regression testing. Despite the
> fact it's very useful, it takes forever, it's easy to make a mistake
> (especially during "reverse regression testing"), users find it too long and
> technical, and only a small minority of regressions are ever bisected. And
> several patches need backporting to allow older versions of Wine to compile
> and run on today's make, gcc, and libraries - this is the case even for the
> 1.0.x releases from less than 3 years ago!
>
> The problem is of course compilation. "configure" takes at least 40
> seconds, without any way to speed it up on multi-core CPUs. "make" takes > 5
> minutes, and it's only taking longer as Wine gets bigger. Compilation is
> fundamentally complex and technical to users.
>
> But what if we had precompiled binaries, and regression testing consisted
> of just running different versions of Wine?
>
> Wine binaries take up about 122 MB and take over 5 minutes to compile.
> There's now 35770 commits between 36def4af0ca85a1d0e66b5207056775bcb3b09ff
> (Release 1.0) and "origin". That's about 4.4 terrabytes of storage and over
> 4 months of compilation, if each of those versions had to be compiled and
> installed into its own prefix, way beyond what most users are willing or
> able to store or do. Most patches however end up affecting only a few binary
> files in the end, and compiling successive versions allows "make" to be very
> quick.
>
> So I've written a tool that compiles Wine and adds each commit's binaries
> into a Git repository. It knows how to compile old versions of Wine
> (currently as far back as 1.0). It knows that commits affecting only
> ANNOUNCE, .gitignore, and files in dll/ or programs/ ending with .c and such
> don't need to go through the endlessly slow "configure", only "make". It is
> stateless: if interrupted, it can resume from the last successful commit. It
> works around bugs in GNU make (you won't believe how many there are...).
>
> This tool compiled all 35000 or so commits from Wine 1.0 to around 4th
> October 2011 in only 7 days, generating a Git repository of Wine binaries
> that's only 26 gigabytes in size. Regression testing with binaries is a
> pleasure: it takes only a few seconds :-) on each bisection. I bisected a 16
> step regression in just 20 minutes, and most of that time was spent running
> the application and dealing with 2 X-server crashes.
>
> I haven't figured out how to make the binaries available to users. Few
> users can clone a 26 gigabyte repository, and even fewer places can serve
> that much to multiple users. Maybe Git can compress it further? The other
> idea I had is that users should be able to regression test through a GUI
> tool. Maybe the GUI tool can just download and run the +/- 122 MB binary
> snapshots for specific commits, instead of having the entire binary
> repository locally?
>
> Any other ideas? Would you like to see this tool? Can I send an attachment
> with it?
>
> Thank you
> Damjan Jovanovic
>
> Hi,

while I agree that 26 GB is a lot for most people, it would be no problem
for me. Long compilation times were exactly the most important factor that
kept me away from testing regressions so far. And I think I am not the only
one on this side, so even if the repository is that huge, I think it might
still worth to grant at least a few people the possibility to clone it.

If I could arrange with my VPS provider to buy more disk space, I would even
be willing to hold a mirror of it online.

I would also definiately like to see how it is all achieved.

Regards,
Ferenc Gergely Szilagyi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20111018/12fa6f6b/attachment.html>