[tools] winetest/dissect: Reject reports with too few test results.

Francois Gouget fgouget at codeweavers.com
Mon Aug 30 12:48:37 CDT 2021

The script does not know how many tests to expect but it can reasonably
expect that $maxfailedtests is only a small fraction of the expected 
number of tests.

Signed-off-by: Francois Gouget <fgouget at codeweavers.com>
In the past WineTest would always show the names of the skipped dlls.
But that made the reports cumbersome for the TestBot (5549ad21b33c) so
when given a list of tests to run (as opposed to a list of tests to
skip), it only prints information about those. As a result dissect does
not know how many tests were skipped (it never really did, it only had
a count of skipped test executables).

It would probably make sense for WineTest to add a line like "skipped N
test executables" in all cases so dissect can better check the report
is complete.

But dissect is a server-side script and thus should not rely on the
client sending truthful and correct information. So I think there is
room for this patch or one like it.

This version of the patch does not guarantee that the report is complete
but it rules out obvious mistakes.

The $maxfailedtests multiplier can be adjusted as desired as long as 
it's less than 10: $maxfailedtests can be up to 50+20 and we have about 
725 test units.

An alternative would be to add a $mintestresults setting in
winetest.conf. It would always be out of date but given that the number
of tests would presumably always go up that should be ok.
Setting it to something large like 700 would work as long as we have
more than 710 tests (700+$maxuserskips; currently we seem to have
about 725).
 winetest/dissect | 1 +
 1 file changed, 1 insertion(+)

diff --git a/winetest/dissect b/winetest/dissect
index 43f7790d2..da4ad636a 100755
--- a/winetest/dissect
+++ b/winetest/dissect
@@ -903,6 +903,7 @@ close_test_unit(1);
 close SUM or mydie "error writing to '$tmpdir/summary.txt': $!";
 close IN;
+mydie "too few tests run (", @boxes-1, ")" if @boxes <= $maxfailedtests * 5;
 mydie "report reached file size limit (runaway test?)" if -s $report >= $maxfilesize;

More information about the wine-devel mailing list