RFC: TestBot priorities

Francois Gouget fgouget at codeweavers.com
Thu Apr 2 21:51:56 CDT 2020


I'm quite set on the number one priority but after that it's more fuzzy. 
So if you feel something's important you can ask for it to be moved up 
the list.


1. Flaky failures blacklisting
   -> Get rid of false positives.
      https://www.winehq.org/pipermail/wine-devel/2020-March/163383.html

2. Use results from timed out WineTest runs
   -> Currently these have to be ignored. But some VMs (e.g. 
      w1064v1809_ja and w1064v1809_hi) time out most of the time so 
      there are not enough reference WineTest results to reliably detect 
      intermittent failures.
      But with test unit blacklisting it would likely be possible to 
      just blacklist intermittent failures so this priority may be 
      lowered (at the cost of already slipping towards a long 
      blacklist). Another alternative would be for Wine developers to 
      fix the timeouts while I do something else.

3. Figure out / fix the occasional long delays in tests
   -> They cause a number of test intermittent failures (timeouts in 
      short WaitForSingleObject() calls). If they can be fixed by 
      tweaking the VM clock configuration then I'd rather figure that 
      out before restoring VMs so I only have one pass to make through 
      them. But initial tests have been inconclusive so if it's going to 
      take long or end up requiring a complex fix like CPU pinning it 
      will have to be delayed.
      https://www.winehq.org/pipermail/wine-devel/2020-March/162846.html

4. Add a Windows 10 1909 VM and rebalance the test configurations.
   -> w1064 has too many configurations to test (7+6) which increases 
      the time it takes to proces jobs. I already created a new w10pro64 
      VM but it's not fully configured yet. Once set up the Windows 10 
      tests will be spread between the vm3 and vm4 hosts, for an ~2x 
      speedup of the Windows 10 tests.

5. Restore the Vista VMs
   -> I've been asked about these a couple of times. So maybe restoring 
      them should be higher priority.

6. Fix the Windows 8 VM WineTest crashes
   -> Most WineTest results are incomplete (but the results are still 
      used which is actually wrong until point 2 is fixed and could 
      cause false positives).

7. Continue fixing miscellaneous issues
   -> Complete automation of the TestBot test suite and then go 
      through it: threaded patch series support, shared Gecko / Mono 
      support, fix WinePrefix update issues, patches site interaction, 
      .com program tests, etc.

-- 
Francois Gouget <fgouget at codeweavers.com>



More information about the wine-devel mailing list