Right way to cope with user error in make test?

[Steven Edwards]

On Mon, May 19, 2008 at 1:47 AM, Dmitry Timoshkov
<dmitry at codeweavers.com> wrote:
> Carefully investigating each test failure one by one as Dan does is
> the right approach IMO, flaming on wine-devel is not.

Sorry if I went a little too flamish in my reply earlier. I did not
mean to imply your point of view was stupid or used the word asinine
it was not addressed at you directly but the general concept that we
have allowed it to be broken for so long.

I offered a proposed solution to a whole class of failures and you
shot it down saying it was the wrong solution for standalone builds. I
addressed that, your answer is now "If some test fails that doen't
mean that the build is broken".  But I don't understand the logic. The
metric should be, if make test fails, the build cannot be assumed
safe. Maybe its not really broken but without a 100% pass rate there
is no way to assume anything other than brokenness. Even standalone is
not a safe test given the current framework as it has 10% failure rate
on Windows Server 2003. Maybe it passes for you on XP, maybe its
broken for me on Vista. Nothing is safe to assume while allow whole
classes of failures. If they really are failures and certain missing
librares makes a build "broken" then it should not be a soft
dependency.

-- 
Steven Edwards

"There is one thing stronger than all the armies in the world, and
that is an idea whose time has come." - Victor Hugo