widl: Add a framework for automated testing of IDL files that should succeed or fail to be parsed.

[Robert Shearman]

Dan Hipschman wrote:
> On Mon, Apr 28, 2008 at 08:37:26PM +0100, Robert Shearman wrote:
>   
>> This should aid in testing more-obscure parts of the parser that aren't 
>> necessarily valid when using RPC (and hence don't make sense being put 
>> in dlls/rpcrt4/tests/server.idl).
>>     
>
> Obviously this is a good idea.  I tried doing something similar a while
> ago, but it didn't get accepted on the first try and I never asked why,
> so I'll ask why now.  See here:
>
> http://winehq.org/pipermail/wine-patches/2006-September/030438.html
>
> It has some benefits over the approach you're proposing so it's worth
> bringing up.  One is that it can run the tests against MIDL, so we know
> the tests themselves are correct (ok, MIDL has been known to do things
> differently than the spec, but we may want to copy MIDL's behaviour
> anyway, and in any case it's easier to automatically validate the tests
> on MIDL than to do so by hand or by inspection).
>   

The above linked patch of yours is what actually inspired me to add 
automated testing of widl. I don't see any reason why we can't run MIDL 
with the same framework. In fact, I think the framework I am proposing 
could better handle this case due to being able to expect specific 
errors/warnings that differ between MIDL and widl (and being able to 
cope with MIDL failing and widl succeeding, for example, due to bugs in 
MIDL).

> Another is versatility in what it can test.  Since the tests are shell
> scripts, not only can it test the success/failure of a parse, but it can
> check that all the correct files were created, and it can perform
> further tests on the output (e.g., if we have an import in the IDL file,
> we can grep the output to make sure a #include was generated for the
> import).
>
> It already supports "todo"s, and the test scripts look similar to the
> usual Wine tests written in C.
>   

Absolutely. The flexibility of your system being able to test the 
contents of files and do other checks for specific tests is a definite 
bonus.

> On the other hand your method has some advantages over mine.  You
> already provide a "make test" framework,

I don't see any reason why your framework can't also be plugged into 
"make test".

> and including the expected
> result of the compilation in the IDL file is cleaner.
>   

Again, that could also be done using your approach. It's just that it 
was necessary with my approach.

> Alexandre may very well disagree with what I see as valid points (like
> testing on MIDL), but these are my suggestions anyway.  Hope they're
> useful.

The way I see it, we have a choice between having a framework that uses 
the makefile to run individual tests of the parser without checking the 
content or a framework that runs every test in one go, but is capable of 
checking the output of the generated files. The only technical advantage 
that I can think of for the former is that it allows the tests to be 
performed in parallel, but I don't know how much of a benefit that is to 
Alexandre (who will be the one doing "make test" the most).

-- 
Rob Shearman