Right way to cope with user error in make test?

[Rolf Kalbermatter]

Please both calm down a bit.

<dmitry at codeweavers.com> wrote:
>"Steven Edwards" <winehacker at gmail.com> wrote:
>><dmitry at codeweavers.com> wrote:
>>> Not everyone supposed to run the Wine tests, only those do who 
>>> *really* can comprehend the results and able to do something with them.
>> 
>> NO. Everyone that builds Wine from source should run make test.
>
>Why? Does it magically make something useful? Again, what about other
>packages built from source?

To limit running of the test frame work only to those that know exactly what
they are doing is a to strict limitation IMHO.

You should not forget that acceptance of a patch is based on the criteria
that it runs make test. Which in may case has been always a limited make
test in the directory itself only, since a whole make test seldom or never
even got so far as to run for the component I was working at.

While sending patches is of course the right thing to do, it has been and
still is rather discouraging to get wine sources, building them on ones own
box and finally run a make test and find that there are many areas where it
simply reports failures or crashes altogether especially in areas I do not
have the knowledge to do anything about and not the time to dig into much
deeper.

The fact that I couldn't get the whole make test to run out of the box on
my system without bad crashes and/or numerous test failures has not only
made me not run it anymore except in tree for the component I attempted
to create a fix for, but has actually discouraged me in the past from
working on more bug fixes or improvements other than what I have found
strictly necessary to make a specific app working.

>> It should pass 100% for everyone everywhere. If does not then there is 
>> something wrong. It does the world not 1 damn bit of good if passes on 
>> your system and no one elses.
>
>If the test doesn't pass for you, go ahead and fix it. I can't do that
>for you if the test passes for me.

But isn't that the crux of why this discussion started? Someone finding
that a soft dependency of Wine on a library should not create a failure
but a skip when that dependency was not satisifed and this was detectable
in a sane way. And getting shot down that this system should be considerd
broken.

If the missing soft dependency should be considered a broken system then
the dependency should be not soft and vice versa.

I personally don't mind a test failure in a library that depends on
something I know I don't use and consequently don't have installed but
if it's possible to skip the tests for that why not do it?

The test framework should not only help avoiding regressions for the
VERY active developers but allow anyone working on Wine in one way or
the other to have some confidence that he got it to build right. It's
what I usually do when installing any source package to see if I missed
something. Getting a long list of errors and sometimes crashes out
of the box really leaves a bad taste and also makes it much harder to
verify that my own modifications to it later on didn't break it in some
ways.

Rolf Kalbermatter