RFC: TestBot priorities
Francois Gouget
fgouget at codeweavers.com
Thu Apr 2 21:51:56 CDT 2020
I'm quite set on the number one priority but after that it's more fuzzy.
So if you feel something's important you can ask for it to be moved up
the list.
1. Flaky failures blacklisting
-> Get rid of false positives.
https://www.winehq.org/pipermail/wine-devel/2020-March/163383.html
2. Use results from timed out WineTest runs
-> Currently these have to be ignored. But some VMs (e.g.
w1064v1809_ja and w1064v1809_hi) time out most of the time so
there are not enough reference WineTest results to reliably detect
intermittent failures.
But with test unit blacklisting it would likely be possible to
just blacklist intermittent failures so this priority may be
lowered (at the cost of already slipping towards a long
blacklist). Another alternative would be for Wine developers to
fix the timeouts while I do something else.
3. Figure out / fix the occasional long delays in tests
-> They cause a number of test intermittent failures (timeouts in
short WaitForSingleObject() calls). If they can be fixed by
tweaking the VM clock configuration then I'd rather figure that
out before restoring VMs so I only have one pass to make through
them. But initial tests have been inconclusive so if it's going to
take long or end up requiring a complex fix like CPU pinning it
will have to be delayed.
https://www.winehq.org/pipermail/wine-devel/2020-March/162846.html
4. Add a Windows 10 1909 VM and rebalance the test configurations.
-> w1064 has too many configurations to test (7+6) which increases
the time it takes to proces jobs. I already created a new w10pro64
VM but it's not fully configured yet. Once set up the Windows 10
tests will be spread between the vm3 and vm4 hosts, for an ~2x
speedup of the Windows 10 tests.
5. Restore the Vista VMs
-> I've been asked about these a couple of times. So maybe restoring
them should be higher priority.
6. Fix the Windows 8 VM WineTest crashes
-> Most WineTest results are incomplete (but the results are still
used which is actually wrong until point 2 is fixed and could
cause false positives).
7. Continue fixing miscellaneous issues
-> Complete automation of the TestBot test suite and then go
through it: threaded patch series support, shared Gecko / Mono
support, fix WinePrefix update issues, patches site interaction,
.com program tests, etc.
--
Francois Gouget <fgouget at codeweavers.com>
More information about the wine-devel
mailing list