Success in todo depending on the Wine platform

[Sebastian Lackner]

On 16.02.2016 00:31, Francois Gouget wrote:
> 
> What should we do if a test succeeds in a todo_wine on Mac but fails on 
> Linux, FreeBSD and Solaris.
> 
> Specifically I'm thinking about this test:
>     /* Check if we have some return values */
>     trace("WorkingSetSize : %ld\n", pvi.WorkingSetSize);
>     todo_wine
>         ok( pvi.WorkingSetSize > 0, "Expected a WorkingSetSize > 0\n");
> 
> Which the following commit caused to succeed on OSX:
> 
> commit 1f9fe124b2bd7e69dad5c90b7897380e32d43481
> Author: Snorri Sturluson <snorri.sturluson at ccpgames.com>
> Date:   Thu Jan 28 17:22:10 2016 +0000
> 
>     ntdll: Fill in memory counters under OS X.
> 
> I find it pretty ironic that the OSX results, which already have more 
> than their fair share or failures, are saddled with one extra failure 
> when it is in fact the only Wine platform where the test succeeds!
> 
> I guess implementing fill_VM_COUNTERS() on Linux, FreeBSD and Solaris 
> would be one solution. It might not even be that hard depending on how 
> correct you want the implementation to be.
> 
> 
> But do we have other solutions? I initially thought setting 
> WINETEST_PLATFORM to Linux, FreeBSD, etc. could do the trick. But it 
> currently it is only ever be set to Windows or Wine. Even worse, 
> todo_wine explicitly checks if it is set to "wine" so anything else 
> would break it. Should todo_wine check for 'not windows' instead?
> 
> 

Changing the definition of todo_wine is not a problem, but in the long term
a more complicated system will get very difficult to maintain.

Its not really feasible to test each commit on all
supported systems {linux,mac,bsd,...} and architectures {x86,x86_64,arm,...}
to find the correct combination of todo_wine_* flags/commands. In some cases,
when the behavior depends on the kernel version, it gets even more complicated.

What we have to keep in mind is that the wine tests are mainly for developers,
and (with minor exceptions) they probably never passed on any other machine
than Alexandres developer box. Adding todo's will not make it better, so we
just have to live with test failures / succeeding tests caused by different
hardware and software, and manually check if its something critical or if it
can be safely ignored.