We *really* need a development model change !

[Andreas Mohr]

On Thu, Dec 27, 2001 at 12:13:05PM -0800, Francois Gouget wrote:
> On Thu, 27 Dec 2001, Andriy Palamarchuk wrote:
> 
> [... Perl/C pros and cons for testing]
> 
>    I think you summarised the pros and cons of both options quite
> well. I would add just one thing against perl in its current form: AFAIK
> (but it may have changed since), the current perl framework does not
> support callback functions. This can be a problem for testing things
> like:
>  * CreateThread: takes a function to be run in the new thread
>  * timers: SetTimer(...,timerproc)
>  * window procs: We can write quite a few tests that create some widget
> (a list, table, ...) and then sends messages to check the state of that
> widget (select an item in the list, then check which item is selected,
> whether the 'hot' item is still hot, etc.), and or the resulting
> sequence of other messages. While these are not 'GUI' tests in that they
> don't make sure that the list displays correctly (or at all), they check
> important properties of the widget implementation.
Not to mention any other "non-easy" tests, such as highly complicated
interaction/message handling/whatever that probably would be rather
problematic to implement in perl.
Thus I guess that we might want to concentrate on a usable C based test
code, while keeping a compatibility scheme for perl based tests in mind.
In short: my test model would be usable from that point of view.
You could concatenate the output of all C programs and perl scripts
easily and get an easy way of determining whether overall compatibility
is healthy.

> > The big question is a tool to test GUI. I did not find
> > any OS Windows GUI testing frameworks :-(
> 
>    Andreas has already replied and i agree with him. But I'll basically
> repeat what he said to give it more weight :-)
> 
>    GUI testing is not 'the big question'. It's irrelevant right now.
>    And what I really don't want to happen, it to see us refuse to pick
> the low hanging fruits and starve to death because we don't have a
> ladder that lets us reach the fruits at the top of the tree.
Yes.
Again, the simple text based "OK"/"FAILED" result would be applicable
easily here, too.
After all it's only the result that matters...
Thus I don't think we need to worry much about it, since most likely
it'll fit into such an infrastructure easily.

>    In other words, there are thousands of APIs that we can test very
> easily and we should write tests for them NOW. We should not wait for
> the development of a hypothetical framework that would let us also test
> graphical (i.e. daoes it display right) issues.
Exactly. The most valuable part of API tests is not their infrastructure,
but rather the fact that they already tested certain functions pretty
well.
E.g. we'd be able to port my test code to a different test framework
terribly easily, as (hopefully ?) all the test conditions of LoadModule16
are known now due to my evaluation of this function.

>    Testing all these APIs represents enough work to keep us busy for
> quite some time and can already benefit Wine greatly. So really, at this
> time, given the amount of effort required to even get something usable,
> "GUI testing" is irrelevant and should be postponed.
My idea exactly. People kept talking about GUI test framework (also at
Codeweavers), but I think we should really get started at all first.

>    (of course if someone out there really wants to develop a GUI testing
> framework and donate it to Wine, go ahead, all I want is that we don't
> wait for one)
Hehe, right :-)

OK, now that we discussed this:
I have to admit that I haven't really looked at the perl winetest yet.
*blush*

I think I'll improve my intended Usenet posting (which will go to
truly relevant Windows devel newsgroups only, I'd say), and then
I'll mail it to interested parties for review.
Any comments/improvements to this text ?

In the meantime we should try to discuss more about what the test suite
framework should look like, i.e. whether my approach is good/bad, what
to possibly improve, what the output should look like and whether
it's suitably parsable.

-- 
Andreas Mohr                        Stauferstr. 6, D-71272 Renningen, Germany
Tel. +49 7159 800604                http://home.nexgo.de/andi.mohr/