Right way to cope with user error in make test?

[Dmitry Timoshkov]

"Steven Edwards" <winehacker at gmail.com> wrote:

> On Sun, May 18, 2008 at 10:02 PM, Dmitry Timoshkov
> <dmitry at codeweavers.com> wrote:
>> How much tests do you usually build, and how often? I do it routinely
>> almost every day on a separate Windows machine by copying single .c
>> files and compiling them as standalone tests. I don't see any rationale
>> behind your suggestion.
> 
> Every time I do a build I try make test.

For what purpose?

> Thats often enough and every
> time it fails and has for years. This should have been unacceptable 5
> years ago and should unacceptable now.

Go ahead and send the patches to fix that. If you can't or don't want to fix
it (for any reason) please stop bitching around, that's at least not productive.
For instance have a look how Dan is doing that.

> Let me try to make it more clear. If the person compiling Wine does
> not have a certain library say libxml2, Wine will still build a stub
> msxml3.dll and build tests but those tests are going to fail when that
> person does make test.

Again, why that person does need/want to run 'make test'? Does that person
run a test suit bundled with each package he/she is building from source?
If yes, for what purpose? Is he/she planning to fix the failures?

> If msxml3 is broken by the lack of libxml2 then
> the tests need to reflect this by skipping.

The person running the test should open his/her eyes and read all the FIXME
messages telling her/him that her/his build is broken. If he/she can't cope
with that, they should leave that job to somebody else, and use prebuilt
Wine binaries.

>> Not everyone supposed to run the Wine tests, only those do who *really*
>> can comprehend the results and able to do something with them.
> 
> NO. Everyone that builds Wine from source should run make test.

Why? Does it magically make something useful? Again, what about other packages
built from source?

> It
> should pass 100% for everyone everywhere. If does not then there is
> something wrong. It does the world not 1 damn bit of good if passes on
> your system and no one elses.

If the test doesn't pass for you, go ahead and fix it. I can't do that for you
if the test passes for me.

>> Again, the tests do not target *anyone*, target audience of Wine tests
>> is Wine developers who is able to cope with the results. Anyone else
>> should play somewhere else.
> 
> Again, if they are building wine from source and smart enough to run
> make, then they are smart enough to run make test or winetest and if
> something fails, they or we need to know about it and know why. We
> should not accept anything less than a 100% pass rate everywhere.

If we == Alexandre, that's a nice assumption. If we == Steven, I think you
should start from trying to do that for your system. This is a developer's
list, right?

>> If you are deveoping a winelib application I don't think you want to run
>> the tests at all. I don't really matter whether the tests pass for you.
> 
> This is asinine. Everyone that has ever developed a Winelib
> application has at some point or another needed to fix some area of
> Wine. If they are working on Wine code then they are going to need to
> run the framework.

That's a too generous assumption. In any case first thing to do is file
a bug report.

>> It doesn't matter how hard we try to stop broken builds, there is not
>> way to stop creating them by an incompetent person. Same true for say
>> kernel builds, and any other project.
> 
> If the build is broken by the result of a certain library then it
> should be a hard dependency and the Wine build should fail. Not the
> bloody testing framework.

If some test fails that doen't mean that the build is broken.

-- 
Dmitry.