Replacing TestLauncher... or not?

[Francois Gouget]

One of the hoped for TestBot improvements is to get rid of TestLauncher 
and replace it with WineTest on the theory that they perform the same 
role.

Indeed both add lines to the test report to identify the start and end 
of each test, and they check that the dlls needed for the test are 
present (which is buggy in TestLauncher).

However WineTest only deals with Wine's test executables and this leads 
to some hardcoded assumptions:
1) It expects to extract the test executables from its own resources.
2) It expects that the executables will be called something like 
   <dllname>_test.exe and that it means it should check if <dllname> is 
   usable.
3) It expects that when given the 'list' parameter the executable will 
   spit out the list of test units to run.
4) Then it expects proceeds to run the test executables with the command 
   line that it chose.

However the TestBot also lets Wine developers upload any executable they 
like and lets them specify the command line they want. This breaks all 
the above assumptions.

So, to replace TestLauncher, WineTest would need to be modified so it 
can run an arbitrary executable, not from its own resources, using an 
arbitrary command line. And it also means merging TestLauncher's code 
that dives into the executable's import table to figure out which dlls 
are needed, with WineTest's code that has support for activation 
contexts (the lack of which is what's causing the bug in TestLauncher if 
I understand correctly).

Finally one would also want to be able to build WineTest without any 
builtin test executable to get something small enough since it would get 
uploaded for each task. And that would still leave all the GUI as dead 
weight. As far as I can tell a WineTest with no builtin test still takes 
around 87 KB whereas TestLauncher can get under 12 KB.

Even in the patch case there are some issues though these have 
essentially been known from the start:
 * Currently it's only possible to build WineTest with all the tests, 
   resulting in a huge executables. Fixing this requires either 
   bypassing the build system to build WineTest, or tweaking the build 
   system so one can specify which executables to embed.

 * Even if we embed only one test executable we usually only want to run 
   one test unit. Fortunately that's already possible by specifying it 
   on the command line.

 * However, even in the case of a patch the Wine developer can specify 
   an arbitrary command line. This is probably not very command but do 
   we want to drop it?

 * Finally WineTest systematically outputs platform information. That 
   would make every report a bit larger and would force developers to 
   skip over it most of the time. But it may also be useful sometimes.


Given the above I feel like it would be simpler, cleaner and possibly 
more maintainable to keep the two separate after all.


Of course sticking with TestLauncher would require some work too but 
that seems quite tractable:
 * First not checking it into Git!
 * Making sure it gets updated automatically.
 * Fixing the dll checks.
 * Bringing it in line with the latest changes w.r.t. pids, run tmie 
   data, etc.


-- 
Francois Gouget <fgouget at codeweavers.com>