TestBot status / Wine todo list
Francois Gouget
fgouget at codeweavers.com
Sun Feb 23 21:57:03 CST 2020
TL;DR;
- WineTest should no longer hit the 1.5 MB report size limit.
- WineTest results have way too many NEW errors (bold orange lines):
https://testbot.winehq.org/JobDetails.pl?Key=65643#k108
Fix them or at least make them not new as shown below.
- There is work for everyone!
* Running WineTest on Wine regularly produced reports above the 1.5 MB
limit.
-> The spam-limiting patch seems to have resolved this issue as no
report has been over the limit since it was committed (e4dbd567).
* WineTest sometimes hits the 30 minutes timeout on Windows which
prevents it from reporting the results to test.winehq.org.
Lately it has mostly impacted the w1064v1809* VMs.
-> Where does WineTest spend time? Why? Can it be sped up?
* There are a number of cases where the TestBot and WineTest either skip
a test because they think some dll is missing, or try to run the test
despite some missing dependency (usually and indirect one), causing a
2 minutes stall.
https://bugs.winehq.org/show_bug.cgi?id=48208
https://bugs.winehq.org/show_bug.cgi?id=31609
* WineTest: get_subtests() fails randomly
https://bugs.winehq.org/show_bug.cgi?id=48061
- Some of these are caused by the previous issue.
- But others are totally random and make no sense like when WineTest
claims that on some days the w1064v1809-hi snapshot does not have
opengl32 or winhttp!
https://test.winehq.org/data/6b35cd2903bc9842ce54cee02caee2106625091d/win10_newtb-w1064v1809-hi-32/version.html
https://test.winehq.org/data/be2b0b1cec5843f0145f376316d6c28507559910/win10_newtb-w1064v1809-hi-32/version.html
-> This causes less trouble now that the TestBot better deals with
intermittent failures. But it still causes some dlls to go
completely untested on some days.
* Fix "always new" failure messages.
- These will cause false positives because the TestBot can never match
them to a previous occurence in the WineTest results.
- To identify them pick any WineTest job and look for new errors.
For instance:
winstation.c:973: Test failed: unexpected foreground window 00000000000A01AC
dxgi.c:5063: Test failed: Got unexpected message 0x31f, hwnd 00000000003E002C, wparam 0x1, lparam 0
- Then check past WineTest reports for that VM configuration, either
on the TestBot or on test.winehq.org and try to figure out why the
failure messaege is different. Usually it will be because the
message contains a handle or pointer value. If that value is not
really important remove it from the failure message.
-> Otherwise some form of the following patch may help:
https://www.winehq.org/pipermail/wine-devel/2019-December/157168.html
https://www.winehq.org/pipermail/wine-devel/2019-December/157169.html
https://bugs.winehq.org/show_bug.cgi?id=48209
* Fix rare intermittent failures / those that never happen in WineTest.
- The TestBot can only identify preexiting failures if they are
present in one of the ~20 past WineTest results it has for each VM
configuration. According to my calculations this means that if the
failure rate is below 6.7% and the failure happens for your patch,
there is a 25% chance the TestBot will blame you.
- Example:
xaudio2.c:67: Test failed: Callbacks called out of order: 1
-> Happened twice in ~60 (newtb*) to ~140 (all) runs.
--
Francois Gouget <fgouget at codeweavers.com>
More information about the wine-devel
mailing list