Improving http://tests.winehq.org/data/

[James Hawkins]

On 10/25/07, Dan Kegel <dank at kegel.com> wrote:
> James wrote:
> >Looking at the test data, all of the msi:install tests timeout.  I
> >just ran the install tests in XP running under vmware on a 3ghz
> >machine.  The tests took 9m41s.  That completely blows away the 2min
> >timeout.  There's nothing wrong with the tests, they just take a long
> >time.   I don't think we should extend the timeout, because it's very
> >subjective and more tests will be added, meaning we'll have to change
> >the timeout eventually.  I do think we should have a flag or variable
> >that allows the timeout to be ignored for certain tests.  Any
> >opinions?
>
> I'd like a way to specify the expected runtime,
> for use on tests that we observe take a long time.
> It would be used in two ways:
> 1) runtest could take an option --skip-long-tests
> which would skip all tests that had that option set, and
> 2) by default runtest would only abort those tests
> if they took one minute over their expected time (say).
>
> That would give us both a quick interactive make test
> and a more reliable slow-but-complete make test.
> - Dan
>

I don't think that's fair to long tests, say msi:install.  There will
always be people that don't want to wait for the tests, and thus the
long tests get less exposure.  In the case of msi:install, there's
technically nothing wrong with the tests, so it should get just as
much exposure as any other test.  Just today I noticed a bug in
msi:install that I didn't know was there before because the tests were
timing out.  Like I said before, we shouldn't specify a limit for
certain tests like msi:install because they'll always get longer and
we'll always be chasing a moving target.

-- 
James Hawkins