Another shot at running tests in parallel

[Dan Kegel]

On modern CPUs, with a 30 second timeout per test,
the wine conformance test suite runs in about 4-5
minutes.  That's wonderfully fast, completely
sufficient for all purposes...

... except one: running tests under valgrind,
which imposes a roughly 10x penalty.
40-50 minutes to run the tests is a bit too long
to expect developers to wait for feedback.

So I had another try at getting conformance tests to run
in parallel, this time without any code modifications.
I use WINETEST_WRAPPER="$HOME/alarm 30",
where alarm is the output of compiling
http://winezeug.googlecode.com/svn/trunk/patchwatcher/alarm.c

That file has a hardcoded whitelist of tests that
are know to not call CreateWindow (directly or indirectly)
or do anything else that might not work well in parallel.
alarm takes an exclusive lock when running
any test not on that list.
Even with this fairly restrictive whitelist, the
tests were fairly parallelizable (see below).

Because running tests in parallel is tricky,
and makes our already somewhat shaky
test suite less stable, I will run the tests
the normal way first, and only if they
pass without regressions will I then do
a second run under Valgrind.  And I'll
ignore any test failures that happen under
valgrind; all I care about in those runs are
the valgrind warnings.

Here are some benchmarks on a quad processor:

-j  time in seconds
1 255
2 155
3 125
4 105
20 79

With -j4, it looks like valgrind could run the
tests in about ten minutes, which would be
quick enough.

It would be nice if our test suite really ran in parallel
some day, but the approximation I have now
is probably good enough.
- Dan

p.s.

Out of curiosity, I checked how many tests interfere
with themselves if multiple copies are run.
To do this, I ran with
WINETEST_WRAPPER="$HOME/foowrap.sh"
where foowrap.sh was

#!/bin/sh
# Run same test three times in parallel
$@ &
sleep 1
$@ &
$@

As you can imagine, that broke a lot of tests,
many more than just running in parallel.