[winetest] Show missing tests in single/group results

[Paul Vriens]

Paul Vriens wrote:
> Hi,
> 
> (I've just remembered that there is a limit on the email size, so I'm 
> doing it a
> bit differently now. The other email, including the screenshots, will 
> probably come through as well)
> 
> Before sending this to wine-patches I would like people to have a look 
> first.
> 
> This patch makes sure that missing tests are shown in the test results, 
> both
> when there is only one system in a group (f.e. Win98) or multiple 
> systems in a
> group.
> 
> On 'http://www.xs4all.nl/~pvriens/200708031000.old.single/' you can see the
> current state of the test results. When a test is missing a '.' is shown 
> (this is of course different from a missing test where we don't have 
> results at all but that's shown as a '.' as well).
> 
> On 'http://www.xs4all.nl/~pvriens/200708031000.old.group/' you can see the
> current state when a group of systems is involved. When tests are 
> missing for a
> group, a '.' is shown with a link to the group overview. There is 
> however no
> missing test output in the group overview (for example gdiplus and pdh)
> 
> 'http://www.xs4all.nl/~pvriens/200708031000.new.single/' shows the new 
> results
> page for single tests. Instead of a single '.' there is a failure shown 
> (red
> box) and a skip border (as these tests have not run). There is still a 
> '.' which
> could maybe better be a '?'. There is also a 'onMouseOver' hint when you 
> go over
> the '.'.
> 
> 'http://www.xs4all.nl/~pvriens/200708031000.new.group/' shows the new 
> results
> when a group is involved. The main summary is still the same as in the 
> old one,
> but the group results now also show the missing tests (see gdiplus and pdh
> again). Again there is a 'onMouseOver' hint in the group results.
> 
> Remember that this is still without winetest producing output about 
> missing dll's.
> 
> Next patches will:
> 
> - add a missing dll as one of the reasons for missing tests
> - add some better clue's on the main summary for multiple systems in a 
> group
> (not a single '.' but maybe the same 'failed and skipped').
> 
> As said, one of the thoughts I had was to replace the '.' with a '?' in 
> case of
> missing tests for an unknown reason but I'm not sure about that yet.
> 
> Comments/remarks etc.. are welcome.
> 
> I'd also like to hear about other idea's for this page to make it useful 
> for
> everybody. Apart from the fact that we probably should have more 
> exposure to the
> fact that we have such a page/tool.
> We saw an immediate rise (slowed down again) in test results as soon as 
> winetest
> was mentioned on 
> 'http://tweakers.net/meuktracker/15817/Wine-0.9.42.html' (in
> Dutch). There it was also mentioned that some machines crashed because 
> of the
> tests etc.. (maybe a wine-winetest component on bugzilla could help with 
> that
> which we could reference on the main test.winehq.org/data/*/index.html 
> pages).
> 
> Cheers,
> 
> Paul.
> 
Forgot to mention that only the index.html file is on the mentioned pages. No
result files (to cut down on size for me to push).

I also forgot to attach the patch. So here it is.

Cheers,

Paul


-------------- next part --------------
--- /wine/tools/winetest/gather	2007-08-03 17:20:09.000000000 +0200
+++ gather	2007-08-05 14:02:42.000000000 +0200
@@ -121,6 +121,22 @@
     push @{$group->{tests}}, $testref;
 }
 
+# Find missing tests. After this exercise all testresults are available
+# for all systems.
+foreach my $group (@groups) {
+    next unless exists $group->{tests};
+    foreach my $test (@{$group->{tests}}) {
+        foreach my $testname (sort keys %alltests) {
+            if (!exists $test->{results}->{$testname}) {
+                # Make sure missing tests are shown in the group results
+                $group->{digests}->{$testname} = "differ";
+                # Mark this test as missing for an unknown reason
+                $test->{results}->{$testname} = ["test missing", "-", "-", "-"];
+            }
+        }
+    }
+}
+
 # Write out the tables
 
 my ($header);                   # same in thead and tfoot
@@ -201,28 +217,30 @@
 # Output a single cell of a test
 sub singletest {
     my ($test, $testname, $groupname) = @_;
-
-    if (!exists $test->{results}->{$testname}) {
-        print OUT "      <td class=\"note\">.</td>\n";
+    my $file = "$test->{dir}/$testname.txt";
+    my ($count, $todo, $error, $skipped) = @{$test->{results}->{$testname}};
+    if ($count eq "failed") {
+        my $msg = $todo eq "-"?"crash":
+                  $todo == 258?"timeout":"failed";
+        my $fail = -r "$datadir/$build/$file"?"<a href=\"$file\">$msg</a>":$msg;
+        print OUT "      <td class=\"note\">$fail</td>\n";
+    } elsif ($count eq "test missing") {
+        print OUT <<"EOF";
+      <td class="skip_fail"><a
+        title="Test did not run for an unknown reason"
+        onMouseOver="Test did not run for an unknown reason";
+        >.</a></td>
+EOF
     } else {
-        my $file = "$test->{dir}/$testname.txt";
-        my ($count, $todo, $error, $skipped) = @{$test->{results}->{$testname}};
-        if ($count eq "failed") {
-            my $msg = $todo eq "-"?"crash":
-                      $todo == 258?"timeout":"failed";
-            my $fail = -r "$datadir/$build/$file"?"<a href=\"$file\">$msg</a>":$msg;
-            print OUT "      <td class=\"note\">$fail</td>\n";
-        } else {
-            my $class = $error?"fail":"pass";
-            my $skip = $skipped?"skip_":"";
-            print OUT <<"EOF";
+        my $class = $error?"fail":"pass";
+        my $skip = $skipped?"skip_":"";
+        print OUT <<"EOF";
       <td class="$skip$class"><a
         href="$file"
         title="$count tests, $todo TODO, $error errors, $skipped skipped"
         onMouseOver="refresh('$testname','$groupname $test->{tag}',$count,$todo,$error,$skipped);"
         >$error</a></td>
 EOF
-        }
     }
 }