Regression testing breakthrough

[Henri Verbeet]

On 18 October 2011 13:42, Damjan Jovanovic <damjan.jov at gmail.com> wrote:
> There's currently another 182 regressions that were closed "ABANDONED".
> Maybe if regression testing was easier and faster, people wouldn't abandon
> them?
>
Maybe. That's 182 closed ABANDONED, out of 2590 total closed, so
that's on the order of 5-10%. Perhaps it's possible to bring that down
by making it easier to do bisects, but it's hardly the case that "only
a small minority of regressions are ever bisected".

> If you are talking about using compiling with ccache instead of the binary
> repository, "configure" alone is > 40 seconds, while the average "git
> bisect" on the binary repository is about 4. If you are talking about using
> ccache to speed up building the binary repository commit by commit, why,
> when for most commits "make" takes about 5 seconds and skips all unnecessary
> compilation anyway?
>
> The other advantage I see is the convenience: instead of having to wait or
> take breaks while each commit compiles like in normal regression testing,
> you can have the tool compile you a small set of revisions (eg. the last
> month) while you are away from your computer, then just quickly test them
> all in one go when you come back. And you can regression test several
> applications without repeatedly compiling Wine.
>
Regardless of whether this has much advantages over e.g. ccache, it
does strike me that this would mostly be useful for people that do a
lot of regression testing already. If it allows those people to do
regressions tests faster and find more regressions or find them sooner
that way that's great, but these aren't the same people that end up
abandoning regressions. The problems there are typically more on the
level of "What is git?" / "How do I build Wine?" / "Where do I get all
these dependencies?", and typically someone would only do one or two
bisects.