Test system for the Widl compiler

[Rémi Bernon]

On 6/15/22 20:04, Zebediah Figura wrote:
> On 6/15/22 12:49, Nikolay Sivov wrote:
>>
>>
>> On 6/15/22 18:20, Bernhard Kölbl wrote:
>>> Hey everyone,
>>>
>>> during the course of implementing WinRT / UWP stuff, I regularly come
>>> across missing IDL 3.0 features in our Widl compiler.
>>> Unfortunately, Widl's code seems to have grown in a rater unpleasant
>>> direction and is sometimes a real burden to understand and maintain. I
>>> think some refactoring to its code would be very well fitting and make
>>> things a lot easier for everyone, but refactoring can also bring
>>> breakage and makes tracking down errors with Git harder. Additionally,
>>> as the code is already not that small, errors have an easy way to slip
>>> in without noticing.
>>> So to me, the ideal solution seems to add a testing system to the
>>> compiler, like most of Wine's components already have.
>>>
>>> Rémi and I already outlined some ideas:
>>> - Port the Winetest system to Unix programs
>>> - Make a libwidl or Widl.exe and test against that with our current 
>>> test system
>>> - Add the tests into Widl itself as callable parameter
>>>
>>> What do you think is the best solution?
>>> Getting some suggestions would be great. :)
>>
>> There are options. First it depends whether you want to validate 
>> against midl or not. If you do then you'll need something like 
>> widl.exe to run on Windows.
>>
>> Next question is what your test data would be, and how to compare the 
>> output reliably with expected output. For example we don't really need 
>> exact matching for generated headers, or even marshaling format 
>> strings, as long as they achieve their purpose. For typelibs however, 
>> it might be useful to go for exact binary match of selected sections.
> 
> We already have tests for typelibs and marshalling, in oleaut32:typelib 
> and oleaut32:tmarshal respectively. I see no reason that's not 
> sufficient already.
> 
> As far as headers are concerned, we could probably take a similar 
> approach in some ways. I don't know what exactly needs testing, but we 
> could do things like
> 
> ok(offsetof(mystruct.field) == 16,
>          "wrong offset %u\n", offsetof(mystruct.field));
> 
> or (for a compile-time assertion)
> 
> extern void test(ITestInterface *obj)
> {
>      HRESULT (*myfunc)(type1 arg1, type2 *arg2, ...);
> 
>      myfunc = obj->lpVtbl->myfunc;
> }
> 
> Ultimately we may not need to mess with widl, or write any extra test 
> infrastructure.
> 

I don't agree and I think it'd be good to have widl parsing tests, to 
validate parser grammar additions, expected parser errors, and the error 
messages we print if we want to make them better. It will improve the 
tool quality and help validating any widl change.

If typelibs and other features are already tested elsewhere, maybe we 
can leave that where it is.

It looks easier to integrate with WineTest and then easier to check the 
changes (compared to custom scripts) if we're doing it with a PE program 
or library. I think a set of tests, linked with a static "idlbase" 
library that implements widl parsing into some IR that the tests would 
validate, would probably be a simple solution.

I don't think we should necessarily aim for midl byte to byte header 
compatibility, but it will also make it possible to have a PE widl.exe 
that could used as a drop-in replacement.

-- 
Rémi Bernon <rbernon at codeweavers.com>