TestBot status / Wine todo list

Francois Gouget fgouget at codeweavers.com
Sun Feb 23 21:57:03 CST 2020


TL;DR;
- WineTest should no longer hit the 1.5 MB report size limit.
- WineTest results have way too many NEW errors (bold orange lines):
  https://testbot.winehq.org/JobDetails.pl?Key=65643#k108
  Fix them or at least make them not new as shown below.
- There is work for everyone!


* Running WineTest on Wine regularly produced reports above the 1.5 MB
  limit.
  -> The spam-limiting patch seems to have resolved this issue as no
     report has been over the limit since it was committed (e4dbd567).

* WineTest sometimes hits the 30 minutes timeout on Windows which
  prevents it from reporting the results to test.winehq.org.
  Lately it has mostly impacted the w1064v1809* VMs.
  -> Where does WineTest spend time? Why? Can it be sped up?

* There are a number of cases where the TestBot and WineTest either skip
  a test because they think some dll is missing, or try to run the test
  despite some missing dependency (usually and indirect one), causing a
  2 minutes stall.
  https://bugs.winehq.org/show_bug.cgi?id=48208
  https://bugs.winehq.org/show_bug.cgi?id=31609

* WineTest: get_subtests() fails randomly
  https://bugs.winehq.org/show_bug.cgi?id=48061
  - Some of these are caused by the previous issue.
  - But others are totally random and make no sense like when WineTest
    claims that on some days the w1064v1809-hi snapshot does not have
    opengl32 or winhttp!
    https://test.winehq.org/data/6b35cd2903bc9842ce54cee02caee2106625091d/win10_newtb-w1064v1809-hi-32/version.html
    https://test.winehq.org/data/be2b0b1cec5843f0145f376316d6c28507559910/win10_newtb-w1064v1809-hi-32/version.html
  -> This causes less trouble now that the TestBot better deals with
     intermittent failures. But it still causes some dlls to go
     completely untested on some days.

* Fix "always new" failure messages.
  - These will cause false positives because the TestBot can never match
    them to a previous occurence in the WineTest results.
  - To identify them pick any WineTest job and look for new errors.
    For instance:
    winstation.c:973: Test failed: unexpected foreground window 00000000000A01AC
    dxgi.c:5063: Test failed: Got unexpected message 0x31f, hwnd 00000000003E002C, wparam 0x1, lparam 0

  - Then check past WineTest reports for that VM configuration, either
    on the TestBot or on test.winehq.org and try to figure out why the
    failure messaege is different. Usually it will be because the
    message contains a handle or pointer value. If that value is not
    really important remove it from the failure message.

  -> Otherwise some form of the following patch may help:
     https://www.winehq.org/pipermail/wine-devel/2019-December/157168.html
     https://www.winehq.org/pipermail/wine-devel/2019-December/157169.html
     https://bugs.winehq.org/show_bug.cgi?id=48209

* Fix rare intermittent failures / those that never happen in WineTest.
  - The TestBot can only identify preexiting failures if they are
    present in one of the ~20 past WineTest results it has for each VM
    configuration. According to my calculations this means that if the
    failure rate is below 6.7% and the failure happens for your patch,
    there is a 25% chance the TestBot will blame you.
  - Example:
    xaudio2.c:67: Test failed: Callbacks called out of order: 1
  -> Happened twice in ~60 (newtb*) to ~140 (all) runs.



-- 
Francois Gouget <fgouget at codeweavers.com>



More information about the wine-devel mailing list