Regression testing breakthrough

[Frédéric Delanoy]

On Wed, Oct 19, 2011 at 14:08, Joel Holdsworth <joel at airwebreathe.org.uk> wrote:
> Alternatively, have you considered doing a .tar.gz of every build snapshot,
> and placing that on a server somewhere?
>
> e.g. a folder full of 36def4af0ca85a1d0e66b5207056775bcb3b09ff.tar.gz files?

tar.xz would compress better

> Then one could write a simple wine regression bisect tool that implements
> similar semantics to git bisect, but would essentially wrap wget. Then in
> your server you could have an index file which is a list of the sha commit
> ids.
>
> This would save the user having to clone a 26Gb repository when most of the
> commits will be irrelevant.

Cloning a multi-gig repository is a no-go for many (most?) people,
especially for a regression testing they might do only once or
twice...
>
> Extra bonus points for doing a better job of compressing the small deltas
> between binaries*, rather than compressing full wine builds.

Maybe you could use stuff like xdelta or bsdiff, but then you may have
some issues IMO:
- (not sure) you should use them on non-compressed files (e.g. .tar)
to get small diffs
- the total size of individual diffs to download could exceed a
single, full, download (especially for old regressions)

To make regression faster/easier, we could (in a script of some sort)
- phase 1: detect the "release range failure" (failed between
wine-1.X.N and wine-1.X.N+1) using only release binaries [or instruct
people to do that first using their distrib packages, like
RegressionTesting does IIRC]
- phase 2: perform bisect between these two releases

Just my 2 ¢

Frédéric