widl: Add a framework for automated testing of IDL files that should succeed or fail to be parsed.
rob at codeweavers.com
Tue Apr 29 08:59:20 CDT 2008
Dan Hipschman wrote:
> On Mon, Apr 28, 2008 at 08:37:26PM +0100, Robert Shearman wrote:
>> This should aid in testing more-obscure parts of the parser that aren't
>> necessarily valid when using RPC (and hence don't make sense being put
>> in dlls/rpcrt4/tests/server.idl).
> Obviously this is a good idea. I tried doing something similar a while
> ago, but it didn't get accepted on the first try and I never asked why,
> so I'll ask why now. See here:
> It has some benefits over the approach you're proposing so it's worth
> bringing up. One is that it can run the tests against MIDL, so we know
> the tests themselves are correct (ok, MIDL has been known to do things
> differently than the spec, but we may want to copy MIDL's behaviour
> anyway, and in any case it's easier to automatically validate the tests
> on MIDL than to do so by hand or by inspection).
The above linked patch of yours is what actually inspired me to add
automated testing of widl. I don't see any reason why we can't run MIDL
with the same framework. In fact, I think the framework I am proposing
could better handle this case due to being able to expect specific
errors/warnings that differ between MIDL and widl (and being able to
cope with MIDL failing and widl succeeding, for example, due to bugs in
> Another is versatility in what it can test. Since the tests are shell
> scripts, not only can it test the success/failure of a parse, but it can
> check that all the correct files were created, and it can perform
> further tests on the output (e.g., if we have an import in the IDL file,
> we can grep the output to make sure a #include was generated for the
> It already supports "todo"s, and the test scripts look similar to the
> usual Wine tests written in C.
Absolutely. The flexibility of your system being able to test the
contents of files and do other checks for specific tests is a definite
> On the other hand your method has some advantages over mine. You
> already provide a "make test" framework,
I don't see any reason why your framework can't also be plugged into
> and including the expected
> result of the compilation in the IDL file is cleaner.
Again, that could also be done using your approach. It's just that it
was necessary with my approach.
> Alexandre may very well disagree with what I see as valid points (like
> testing on MIDL), but these are my suggestions anyway. Hope they're
The way I see it, we have a choice between having a framework that uses
the makefile to run individual tests of the parser without checking the
content or a framework that runs every test in one go, but is capable of
checking the output of the generated files. The only technical advantage
that I can think of for the former is that it allows the tests to be
performed in parallel, but I don't know how much of a benefit that is to
Alexandre (who will be the one doing "make test" the most).
More information about the wine-devel