TestBot News

[Francois Gouget]

On Wed, 4 May 2022, Rémi Bernon wrote:
[...]
> When a patch is submitted that is detected as potentially touching more than a
> single test, all the tests for the module are queued for testing. However this
> isn't done through WineTest, and instead they are all queued and tested
> separately, at least on the Windows VMs.
> 
> Wouldn't it be better to always run the tests through WineTest, and make it
> run all the tests that need checking at once?

That would certainly be more efficient time-wise.

Network-wise there's the issue that WineTest.exe is big because it 
always contains all the tests so it would cause more network traffic. 
But the traffic issue is probably minor and it is probably possible to 
tweak the builds to reduce the size.

But as you mentionned, the main issue is that the tests could interfere 
with each other which so far has been regarded as "polluting" the 
results. But we could see things differently.


> Another question, unrelated to the performance problems, could we consider
> adding more Desktop/WM environments to the Debian VMs? I think it could be
> interesting to have to track down winex11 bugs, though it's probably likely to
> have several broken tests.

So far the main goal has been to avoid failures so the desktop 
environment has been optimized with that in mind (so fvwm with a 
carefully crafted configuration).

But again things have changed since then. Most importantly the TestBot 
can now distinguish old failures from new ones and I'm still working 
towards having a way to prevent the "always new" failures from causing 
false positives.

With those two in place running the tests in configurations known to 
cause failures is less of an issue.

One way support for multiple desktop environments could be done in 
the current framework would be to have one Linux VM per desktop 
environment. However that means compiling once per test environment 
which has an impact on performance. With a fast new server (or servers) 
that could work though.

The alternative would be to install multiple desktop environments in the 
same Debian VM (easy) and have the client-side TestBot script switch 
from one desktop environment to another based on the configuration to 
test. I'm not sure how that would work though.


> I also intend at some point, when win32u conversion will be more settled, to
> finish sending my nulldrv patches, and I think it'd be nice to have a testbot
> flavor that could be configured to use it instead of the default graphics
> driver.

It sounds like that's just a matter of configuring the test environment 
to use nulldrv instead of the regular graphics driver (including 
possibly unsetting $DISPLAY). So that would be a bit like setting the 
locale and could probably be done through the missions mechanism without 
requiring a separate test environment.


-- 
Francois Gouget <fgouget at codeweavers.com>