[tools] testbot: Increase the full test suite timeout to 45 minutes.

Francois Gouget fgouget at codeweavers.com
Mon Feb 8 15:07:07 CST 2021


30 minutes is really too short to run the 700+ tests nowadays.

Signed-off-by: Francois Gouget <fgouget at codeweavers.com>
---

Here are the average 32-bit WineTest run times per host (not counting 
wineprefix creation and file transfers):
cw-gtx560     36.1 mn
cw-rx460      30.2 mn
debiant       34.5 mn
debiant2      43.6 mn
fg-debian10   34.5 mn

Even though the top 13 tests account for 50% of the current 30 minute 
timeout I doubt we'll get the TestBot VMs run time significantly down.

Most of the 13 are 3D tests and so the suffer from the lack of 
acceleration or, maybe, of X11 1.7.0 bugs. The 3 exceptions are 
cmd.exe:batch and mlang:mlang (which almost don't touch the screen but 
still manage to be slow); and user32:msg.

For reference here is a breakdown:
Test            Time    Cumulative % of 30 mn
cmd.exe:batch   120.0 s 6.7%
user32:monitor  99.8 s  12.2%
dxgi:dxgi       78.6 s  16.6%
d3d9:device     75.4 s  20.8%
mlang:mlang     73.2 s  24.8%
ddraw:ddraw7    73.0 s  28.9%
ddraw:ddraw4    72.0 s  32.9%
d3d11:d3d11     71.2 s  36.8%
ddraw:ddraw2    62.2 s  40.3%
user32:msg      55.6 s  43.4%
d3d9:visual     54.4 s  46.4%
ddraw:ddraw1    54.0 s  49.4%
d3d8:device     49.8 s  52.2%

The top 8 tests that push debiant2 higher than debiant are:

Test                    Additional time
dwrite:font             +84 s
regedit.exe:regedit     +58 s
dwrite:analyzer         +55 s
dwrite:layout           +49 s
reg.exe:reg             +29 s
dxgi:dxgi               +23 s
shell32:shlexec         +22 s
comctl32:status         +19 s

* The 3 dwrite tests are because debiant2 has more fonts for the 
  Chinese, Japanese and other locales.
* The regedit and reg tests might be related to that too.
* The remainder are a mystery though debiant2 has X11 1.7.0 instead of 
  X11 1.6.7 so that may be a factor for some tests.

If you want to dig, you can get all the data at the URL below:
http://fgouget.free.fr/tmp/winetest-times-20210205.xls


 testbot/lib/WineTestBot/Config.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/testbot/lib/WineTestBot/Config.pm b/testbot/lib/WineTestBot/Config.pm
index 2f01e951b..312797229 100644
--- a/testbot/lib/WineTestBot/Config.pm
+++ b/testbot/lib/WineTestBot/Config.pm
@@ -130,7 +130,7 @@ $TimeoutMargin = 2 * 60;
 
 # Test timeouts (in seconds)
 # - For the whole test suite
-$SuiteTimeout = 30 * 60;
+$SuiteTimeout = 45 * 60;
 # - For the first two tests
 $SingleTimeout = 2 * 60;
 # - For extra tests
-- 
2.20.1



More information about the wine-devel mailing list