Do we need a Mr Tests?

[Francois Gouget]

On Sat, 5 Oct 2019, Sven Baars wrote:
[...]
> so it must be due to some testbot change. I was actually under the
> assumption that you were checking the test log after testbot changes to
> see if the changes caused any failures, but I guess not.

That depends on the type of change. Most of the TestBot changes (commits 
to tool/testbot) have no impact on the test results. Only the VM changes 
would make a difference. So I normally check the results after changing 
the VMs. But it's not actually very practical because when VMs already 
have a significant number of failures, some of which are random, it's hard 
to say if a new run is really better or worse :-(


[...]
> Since I already look at it quite often, I can just file bug reports when
> I see consistent new failures if you like.

Yes, I think that could be useful. In particular if the new failures are 
caused by changes in the tests it's useful to identify the bad commit, or 
at least the range of bad commits.

If the new failure is not caused by a change in the tests and there's no 
other obvious explanation, then maybe it's better to ask what's up on 
wine-dev.


-- 
Francois Gouget <fgouget at free.fr>