RFC: 'test' bugzilla keyword

[Paul Millar]

Hi everyone,

A few thoughts ... although the bits about winelib should be taken with a 
pinch of salt: I've never used winelib in anger.

Personally, I don't like the keyword "test". Its far too ambiguous.

On Tue, 27 Aug 2002, Francois Gouget wrote:
> Maybe 'conformance' would work?

Personally, I prefer "regression" to "conformance", although both are
applicable names.

Yes, the tests are checking that wine _conforms_ to what we've specified
(via the tests) as correct windows behaviour, but equally if a test fails
then wine has _regressed_ to a more primitive (and therefore less useful)
state.

I think most people would think of "regression testing" as an automated
process: a suite of many (usually short) tests all of which which must
pass. This is just "make testclean && make test || echo Regression found" 
:)

The problem (in terms of nomenclature) is that wine (as opposed to
winelib) runs applications. So a regression is also when an application
doesn't behave as well as it did. (This is in the docs as what we mean by
"regression".)  But, from another point-of-view, running an application is
just another test. Its a test that checks _lots_ of different aspects of
wine.

Perhaps another possibility worth considering is winelib: whether test
applications continue to build correctly. If something were to interfere
with winelib (perhaps subtle enough not to affect wine running of native
applications), then a different type of regression has been introduced.

So I'd say we have possibility of regressions in:
   1. core Windows functionality - currently tested by many "little" 
                                   tests: an automated process.
   2. running native applications - tested by people using wine and 
                                    complaining when something breaks.
   3. winelib infrastructure - tested by building tools? People 
                               complaining?

If we cannot sub-categorise "regression" in bugzilla, how about implicitly
subgrouping. For example, calling them something like:
  regression-automated
  regression-application
  regression-winelib

Alternatively, we could have the "regression" keyword and also the
"application", "winelib", "automated" keywords. The bug-reporter adds 
whatever s/he feels best describes their bug.

Cheers,

Paul.

----
Paul Millar