Regression testing breakthrough

[Damjan Jovanovic]

On Tue, Oct 18, 2011 at 12:08 PM, Henri Verbeet <hverbeet at gmail.com> wrote:

> On 18 October 2011 10:45, Damjan Jovanovic <damjan.jov at gmail.com> wrote:
> > (especially during "reverse regression testing"), users find it too long
> and
> > technical, and only a small minority of regressions are ever bisected.
> And
> Not true. Even for the regressions that are still open it's currently
> 276 bisected vs. 99 not bisected, and Alexandre said about 90% of
> opened regressions get fixed.
>
>
There's currently another 182 regressions that were closed "ABANDONED".
Maybe if regression testing was easier and faster, people wouldn't abandon
them?


> > I haven't figured out how to make the binaries available to users. Few
> users
> > can clone a 26 gigabyte repository, and even fewer places can serve that
> > much to multiple users. Maybe Git can compress it further? The other idea
> I
> > had is that users should be able to regression test through a GUI tool.
> > Maybe the GUI tool can just download and run the +/- 122 MB binary
> snapshots
> > for specific commits, instead of having the entire binary repository
> > locally?
> >
> > Any other ideas? Would you like to see this tool? Can I send an
> attachment
> > with it?
> >
> Scripts for making old versions compile may be useful, but for the
> most part it sounds like you're essentially duplicating ccache.
>

If you are talking about using compiling with ccache instead of the binary
repository, "configure" alone is > 40 seconds, while the average "git
bisect" on the binary repository is about 4. If you are talking about using
ccache to speed up building the binary repository commit by commit, why,
when for most commits "make" takes about 5 seconds and skips all unnecessary
compilation anyway?

The other advantage I see is the convenience: instead of having to wait or
take breaks while each commit compiles like in normal regression testing,
you can have the tool compile you a small set of revisions (eg. the last
month) while you are away from your computer, then just quickly test them
all in one go when you come back. And you can regression test several
applications without repeatedly compiling Wine.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.winehq.org/pipermail/wine-devel/attachments/20111018/3c73ae1e/attachment.html>