TestBot News

[Zebediah Figura (she/her)]

On 5/5/22 10:37, Francois Gouget wrote:
> On Wed, 4 May 2022, Zebediah Figura (she/her) wrote:
> [...]
>> This seems worth looking into.
> 
> Given that almost all the CPU performance gains come from high core
> counts nowadays I agree that it would be nice. But...
> 
> 
>> There's also a lot of tests that can't really be improved by
>> multithreading *internally*, but also don't touch global state and
>> hence could be run in parallel with anything else. We could construct
>> a whitelist (or maybe there's even enough to construct a blacklist
>> instead) of tests that winetest can run in parallel.
> 
> I don't think there's a way to automatically detect which test units can
> be run in parallel or even to have a heuristic that reliably identifies
> a subset that are safe to parallelize. (and reciprocally for a
> whitelist)
> 
> That means we'd need a handcrafted whitelist or blacklist and I'm not
> sure how maintainable that would be:
> - A blacklist has the drawback that we'd always be playing catchup to
>    add new tests.
> - But I'm not even sure a whitelist would work better: any patch to a
>    whitelisted test may require evicting it from the whitelist.
> 
> 

Obviously it'd have to be maintained manually, but my idea is that we 
can work towards a blacklist, and try to make it as small as possible, 
probably by gradually making tests parallelizable, and then by adding 
the requirement that new changes to tests avoid breaking that. Which I 
don't think is an unreasonable requirement to have or enforce.

Ultimately I don't think it'd be that bad, either.

Off the top of my head, tests I can think of that inherently can't be 
parallelized:

* MSI installation tests (msi:action and msi:install, although not 
msi:db and msi:format); Windows only allows one installer to be run at once.

* dinput and ntoskrnl tests, I think? Probably also setupapi? I'm not 
sure these couldn't be made independent of each other, but it's probably 
easier not to try.

* Tests which change display mode (some ddraw, d3d8, d3d9, dxgi, 
user32:sysparams). In many cases these are put into test units with 
other d3d tests which *are* parallelizable, but they could be split out. 
Although, that said:

* d3d tests in general are an odd case. We can't parallelize them if we 
might run out of GPU memory, although that hasn't been a concern yet and 
it won't be for llvmpipe. We also can't parallelize them on nouveau 
because of its threading problems. There are also a few tests that 
*shouldn't* break other tests but do because of driver bugs.

* Tests which care about the foreground window. In practice this 
includes some user32, d3d, dinput tests, probably others. Often it's 
only a couple of tests functions out of the whole test. (I wonder if we 
could improve things by creating custom window stations or desktops in 
many cases?)

* Tests which warp the cursor or depend on cursor position. This ends up 
being about the same set.

A quick skim doesn't bring up any other clear cases of tests that can't 
be parallelized. There are probably still a lot that need auditing and 
perhaps extra work to ensure that they can be parallelized, but I think 
that's work worth doing.

There are a decent number of tests that hardcode temporary file paths 
but could be made to use GetTempFileName() instead. Actually most such 
tests already use GetTempFileName(), I guess in order to be robust.

There's also a lot of tests that do touch global state, e.g. write to 
the registry, but don't write to parts of the registry that any other 
tests should be reading. advapi32:registry is one such example. Such 
tests can't run in parallel with *themselves*, but could be run in 
parallel with anything else.

(Stuff like services.exe or advapi32:service might fall into the same 
boat. These do touch global state, but in theory other tests shouldn't 
care that we're e.g. creating an advapi32 test service. Definitely 
easier just to blacklist those, though, at least to start with...)

One other thing that occurs to me while writing this is that instead of 
using a blacklist in winetest.exe, we could use global (win32) mutex 
objects in the relevant tests. That would also allow us to separate 
tests which can't be paralellized only with a set group of other tests 
(and run e.g. msi:action and d3d8:device at the same time), as well as 
have finer grained control than blacklisting a whole test file (e.g. we 
could grab WINETEST_DISPLAY_MODE_MUTEX around only the d3d8:device tests 
that mess with the display mode).