Regression testing breakthrough

[Hin-Tak Leung]

I have two suggestions - 

- git clone has a "--depth" option which does a shallow clone (i.e. with some history removed).

- you can use "git-archive" to export arbitrary commits out as a tar ball dynamically; there is no need to have store a tar ball permanently.

That said, I am doubtful about the usefulness of this in a linux environment, because the system libraries (in the general sense - libjpeg, etc) varies between different linux distros.

--- On Wed, 19/10/11, Joel Holdsworth <joel at airwebreathe.org.uk> wrote:
    
    Alternatively, have you considered doing a .tar.gz of every build snapshot, and placing that on a server somewhere?
  
   e.g. a folder full of
   36def4af0ca85a1d0e66b5207056775bcb3b09ff.tar.gz files?
   
  Then one could write a simple wine regression bisect tool that implements similar semantics to git bisect, but would essentially wrap wget. Then in your server you could have an index file which is a list of the sha commit ids.
   
  This would save the user having to clone a 26Gb repository when most of the commits will be irrelevant.
   
  Extra bonus points for doing a better job of compressing the small deltas between binaries*, rather than compressing full wine builds.
   
  Joel
   
  * Are binaries deterministic like this? or do they tend to be completely scrambled?  

   On 18 October 2011 at 09:45 Damjan Jovanovic <damjan.jov at gmail.com> wrote:

   > Hi
   >
   > Since the beginning, I've had issues with regression testing. Despite the
   > fact it's very useful, it takes forever, it's easy to make a mistake
   > (especially during "reverse regression testing"), users find it too long and
   > technical, and only a small minority of regressions are ever bisected. And
   > several patches need backporting to allow older versions of Wine to compile
   > and run on today's make, gcc, and libraries - this is the case even for the
   > 1.0.x releases from less than 3 years ago!
   >
   > The problem is of course compilation. "configure" takes at least 40 seconds,
   > without any way to speed it up on multi-core CPUs. "make" takes > 5 minutes,
   > and it's only taking longer as Wine gets bigger. Compilation is
   > fundamentally complex and technical to users.
   >
   > But what if we had precompiled binaries, and regression testing consisted of
   > just running different versions of Wine?
   >
   > Wine binaries take up about 122 MB and take over 5 minutes to compile.
   > There's now 35770 commits between 36def4af0ca85a1d0e66b5207056775bcb3b09ff
   > (Release 1.0) and "origin". That's about 4.4 terrabytes of storage and over
   > 4 months of compilation, if each of those versions had to be compiled and
   > installed into its own prefix, way beyond what most users are willing or
   > able to store or do. Most patches however end up affecting only a few binary
   > files in the end, and compiling successive versions allows "make" to be very
   > quick.
   >
   > So I've written a tool that compiles Wine and adds each commit's binaries
   > into a Git repository. It knows how to compile old versions of Wine
   > (currently as far back as 1.0). It knows that commits affecting only
   > ANNOUNCE, .gitignore, and files in dll/ or programs/ ending with .c and such
   > don't need to go through the endlessly slow "configure", only "make". It is
   > stateless: if interrupted, it can resume from the last successful commit. It
   > works around bugs in GNU make (you won't believe how many there are...).
   >
   > This tool compiled all 35000 or so commits from Wine 1.0 to around 4th
   > October 2011 in only 7 days, generating a Git repository of Wine binaries
   > that's only 26 gigabytes in size. Regression testing with binaries is a
   > pleasure: it takes only a few seconds :-) on each bisection. I bisected a 16
   > step regression in just 20 minutes, and most of that time was spent running
   > the application and dealing with 2 X-server crashes.
   >
   > I haven't figured out how to make the binaries available to users. Few users
   > can clone a 26 gigabyte repository, and even fewer places can serve that
   > much to multiple users. Maybe Git can compress it further? The other idea I
   > had is that users should be able to regression test through a GUI tool.
   > Maybe the GUI tool can just download and run the +/- 122 MB binary snapshots
   > for specific commits, instead of having the entire binary repository
   > locally?
   >
   > Any other ideas? Would you like to see this tool? Can I send an attachment
   > with it?
   >
   > Thank you
   > Damjan Jovanovic