Right way to cope with user error in make test?

[Steven Edwards]

On Sun, May 18, 2008 at 10:02 PM, Dmitry Timoshkov
<dmitry at codeweavers.com> wrote:
> How much tests do you usually build, and how often? I do it routinely
> almost every day on a separate Windows machine by copying single .c
> files and compiling them as standalone tests. I don't see any rationale
> behind your suggestion.

Every time I do a build I try make test. Thats often enough and every
time it fails and has for years. This should have been unacceptable 5
years ago and should unacceptable now. I don't care what causes the
failure, if its a broken third party application like metacity or a
soft dependency missing. This complacency has kept the framework from
passing 100% for anyone on anything other than Alexandre's box. At the
very least we should have targeted a certain version of Windows years
ago and said "It is unacceptable for the framework to ever fail here"
but I digress. To return to your question, as I am rejoining the
ReactOS project on June 1st I plan on doing a lot more work with
winetest as per Alexandre's statement years ago that ReactOS
contributions to the test suite would be allowed.

Let me try to make it more clear. If the person compiling Wine does
not have a certain library say libxml2, Wine will still build a stub
msxml3.dll and build tests but those tests are going to fail when that
person does make test. If msxml3 is broken by the lack of libxml2 then
the tests need to reflect this by skipping.

> Not everyone supposed to run the Wine tests, only those do who *really*
> can comprehend the results and able to do something with them.

NO. Everyone that builds Wine from source should run make test. It
should pass 100% for everyone everywhere. If does not then there is
something wrong. It does the world not 1 damn bit of good if passes on
your system and no one elses.

> Again, the tests do not target *anyone*, target audience of Wine tests
> is Wine developers who is able to cope with the results. Anyone else
> should play somewhere else.

Again, if they are building wine from source and smart enough to run
make, then they are smart enough to run make test or winetest and if
something fails, they or we need to know about it and know why. We
should not accept anything less than a 100% pass rate everywhere.

> If you are deveoping a winelib application I don't think you want to run
> the tests at all. I don't really matter whether the tests pass for you.

This is asinine. Everyone that has ever developed a Winelib
application has at some point or another needed to fix some area of
Wine. If they are working on Wine code then they are going to need to
run the framework.

> It doesn't matter how hard we try to stop broken builds, there is not
> way to stop creating them by an incompetent person. Same true for say
> kernel builds, and any other project.

If the build is broken by the result of a certain library then it
should be a hard dependency and the Wine build should fail. Not the
bloody testing framework.

-- 
Steven Edwards

"There is one thing stronger than all the armies in the world, and
that is an idea whose time has come." - Victor Hugo