We *really* need a development model change !

[Francois Gouget]

On 2 Jan 2002, Alexandre Julliard wrote:

> Francois Gouget <fgouget at free.fr> writes:
[...]
> IMO the test should not be printing successes or failures at all. If
> it can determine whether some result is OK or not, it should simply do
> an assert on the result. Printing things should be only for the cases
> where checking for failure is too complicated, and so we need to rely
> on the output comparison to detect failures.

   I am not sure about using asserts. In the case where one calls the
same API ten times with different parameters, it would probably be
better to still do all ten tests even if the second call fails. This way
the developper debugging things immediately knows if just one aspect
failed or if its everything. If we call assert as soon as one item fails
we don't have the information of whether the rest still works or not. Of
course this assumes that a failure is not fatal for the rest of the
tests. If it is, then an assert could make sense.
   But I agree tests could be written so as not to print anything unless
there is a failure. Especially for simple cases like checking return
codes, there is no need to print the return code and to check it against
a reference file. Then a single 'Test: Ok' before the final return would
make sure we don't get an empty output file if that is important.


> >    Otherwise this file is either called:
> >  * 'xxx.ref'
> >  * or 'xxx.win95' or 'xxx.win98' ... if the output depends on the
> > Windows version being emulated. The winever-specific file takes
> > precedence over the '.ref' file, and the '.ref' file, which should
> > exist, serves as a fallback.
>
> No, there should always be a single .ref file IMO. Version checks
> should be done inside the test itself to make sure the output is
> always the same.

   The same way there are cases where the simplest solution for checking
the result is correct is to compare it against a reference file, I'm
thinking that there are cases where the somplest way to make sure we
behave the same way as a specific windows version is to compare the
output to that windows version. But I must admit that I don't have such
an example for now so we can start with the single '.ref' file for now.
Then, if necessary, the above mechanism should be relatively simple to
put in place.


> >    Each test should contain a section that looks something like:
> >
> > # @START_TEST_COVERAGE@
> > # kernel32.CreateProcess
> > # kernel32.GetCommandLineA
> > # kernel32.GetCommandLineW
> > # __getmainargs
> > # __p__argc
> > # __p_argv
> > # __p_wargv
> > # @END_TEST_COVERAGE@
>
> This is already part of the Perl framework, you have to explicitly
> declare the functions you use. So we don't want to duplicate the
> information.

   But in the case where a test has to call GetVersion() to handle
differences between windows versions, we surely don't want to go on
record as saying that this test tests GetVersion.
   Similarly, pretty much all GUI tests will call things like
RegisterClassExA, CreateWindowExA, GetMessageA, ... then, finding the
two or three tests that really test these will be pretty hard.
   Maybe the test framework could have an alias for "wine::declare",
maybe "wine::test". Then APIs declared with "wine::test" are those that
are being tested while those declared with "wine::declare" are just
overhead.


> >  * TEST_WINVER
> >    This contains the value of the '-winver' Wine argument, or the
> > Windows version if the test is being run in Windows. We should mandate
> > the use of specific values of winver so that test don't have to
> > recognize all the synonyms of win2000 (nt2k...), etc.
> >    (Do we need to distinguish between Windows and Wine?)
>
> The test should use GetVersion() and friends IMO, no need for a
> separate variable.

   Good point.


> >  * and add two corresponding targets: 'make cui-tests' runs only those
> > tests that do not pop up windows, and 'make gui-tests' runs only those
> > tests that do pop up windows
> >  * 'make tests' would be 'tests: cui-tests gui-tests'
>
> I don't think this complexity is necessary. You can always redirect
> the display if the windows annoy you. And tests should try to keep the
> windows hidden as much as possible.

   Ok. I guess we can still split tests in two categories later if
necessary.


> >    What is needed most is a two sample tests:
> >  * one simple console based test
> >  * another one involving some GUI stuff
>
> I have attached two sample Perl scripts that were written some time
> ago by John Sturtz and myself. One is testing the atom functions and
> the other is creating a window. They should probably be simplified a
> bit to serve as documentation samples.

   Nice :-)


--
Francois Gouget         fgouget at free.fr        http://fgouget.free.fr/
                         "Utilisateur" (nom commun) :
        Mot utilisé par les informaticiens en lieu et place d'"idiot".