[test.winehq.org] missing tests

[Ferenc Wagner]

"Dimitrie O. Paun" <dpaun at rogers.com> writes:

> Speaking of the test results, I've noticed the following problems:
>   1. Some errors reported in the summary 
>      don't get reported in the differences.

Good catch, fixed (*)

>   2. The differences tables are inconsistent.

How can you say that?!  Do you think WineHQ can't reliably
run my program?  :)  You may find my logic wrong, though.
The differences show, well, the differences.  Lines are
pruned iff - all the reports are the same (char by char) AND
           - no run failed AND
           - there were no errors (* from now).
It doesn't work as well as I expected because lots of
successful tests produce variable output.  I thought about
fixing them, but I could as well change the pruning policy
with much less work.  Shall I simply drop the first condition?

>      we also shouldn't make the '0' in the summary a link,
>      since it has nowhere to take us.

Those links don't target specific lines, so they could as
well stay, but I've got no objection against nuking them.

>   3. We are running multiple test _per_ build, but only
>      one is curretly reported. Currently, it says:
> 	Main summary for build 200406171000 
>      where '200406171000' is a link to the test. But since
>      we have multiple downloads, it should be:
> 	Main summary for build 200406171000: [0] [1] [2] [3]
>      (or somesuch), where [x] is a link to the download.

This is a more complicated issue which I haven't dared to
touch yet.  I planned to handle this as different builds, so
there is no machinery in place for this variability.
Neither for the download link, nor for the results.  The Tag
could be overloaded for this purpose, and the download links
should go into the differences header for easy access, where
they could also serve as a quick indication of the build
type.  Then we need short names or icons for the builds.

>   4. Each of the testers run and submit 4 sets of tests for
>      a build, but only one is reflected in the results.
>      This is related to the above point, and needs fixing.

4?  I though it was 2: Kevin's and Paul's.  Or do they both
plan to build with MinGW and MSVC?  Anyway, the reports
should be present with the same tag, successively numbered.
The annoying thing is that people started to number their
tags, which leads to strange-looking headers.  Oh well.

It would be possible to subdivide the columns under the tags
by builds.  We would need a short build id for this at a
well-defined place, like in a new field in the report or
maybe in the first line of the Build info: block.
-- 
Feri.