TestBot News

[Francois Gouget]

On Thu, 5 May 2022, Zebediah Figura (she/her) wrote:
[...]
> Off the top of my head, tests I can think of that inherently can't be
> parallelized:
[...]
> * Tests which change display mode (some ddraw, d3d8, d3d9, dxgi,
> user32:sysparams). In many cases these are put into test units with other d3d
> tests which *are* parallelizable, but they could be split out.

I would add user32:monitor.


[...]
> * d3d tests in general are an odd case. We can't parallelize them if we might
> run out of GPU memory, although that hasn't been a concern yet and it won't be
> for llvmpipe.

Do we really use that much GPU memory?


> We also can't parallelize them on nouveau because of its threading 
> problems. There are also a few tests that *shouldn't* break other 
> tests but do because of driver bugs.

The resolution change tests always leave my monitor in a weird 
resolution like 320x200 when it's not 200x320 (portait mode). It's 
always fun in the morning to find a terminal to issue an xrandr -s 0. 
But I suspect the first WineTest (win32) run may break the next WineTest 
run (wow64) whenever a test tries to open a window that does not fit in 
that weird desktop resolution. I suspect comctl32:combo, header, rebar, 
status and toolbar are among the impacted tests. (so I'm now trying to 
inject an xrandr in between runs)

All that to say that if the resolution change tests run in parallel or 
at a somewhat random time relative to the other tests that may bring 
more variability and unexpected failureto the results.


> * Tests which care about the foreground window. In practice this includes some
> user32, d3d, dinput tests, probably others. Often it's only a couple of tests
> functions out of the whole test. (I wonder if we could improve things by
> creating custom window stations or desktops in many cases?)
>
> * Tests which warp the cursor or depend on cursor position. This ends up being
> about the same set.

I may be wrong but I suspect this should include most of comctl32, 
comdlg32, user32:edit, and probably others.


> A quick skim doesn't bring up any other clear cases of tests that can't be
> parallelized. There are probably still a lot that need auditing and perhaps
> extra work to ensure that they can be parallelized, but I think that's work
> worth doing.

There's also all the timing issues in sound, locking (timeout aspects) 
and timer tests.


> There are a decent number of tests that hardcode temporary file paths but
> could be made to use GetTempFileName() instead. Actually most such tests
> already use GetTempFileName(), I guess in order to be robust.

Eh, funny you should say that. I just found out that kernelbase:process 
and lz32:lzexpand_main forgot to do that (bug 52970). But yes, easily 
fixable.


But overall I'm more skeptical about the feasibility of parallelization. 
For instance the w10pro6v2004 and w10pro64 test configurations had a 
background Windows Update causing failures in msi:msi and msi:package. 
So far quite understandable. But that also caused reproducible failures 
in ieframe:webbrowser, kernel32:resource, shell32:shlfileop, urlmon:url, 
wininet:http and wininet:urlcache (bug 52560). That's kind of wide 
ranging and unexpected.

Maybe it could work by only letting one test to run in each of very 
very broad categories (maybe that's similar to your CS idea):
  * screen : anything that opens a window or modifies the screen
        d3d*, user32*, gdi32*, etc.
  * sound : anything that plays or captures sound
        dsound, winmm, mmdevapi, etc.
  * timing : anything sensitive to timing
        dsound, winmm, mmdevapi, kernel32:sync, etc.
  * install
        msi*, ntoskrnl*, more?
  * others : anything not in any of the above categories

But even such a scheme would probably allow msi:msi to run in parallel 
with urlmon:url and bug 52560 seems to indicate that would not be a good 
idea.

Also I'm not sure we'd have much parallelism left with such a scheme 
(i.e. too much complexity for too little gain?).

But also maybe the only way to know is to try.

-- 
Francois Gouget <fgouget at codeweavers.com>