Do we need a Mr Tests?

[Sven Baars]

On 05-10-19 02:02, Francois Gouget wrote:
> 
> I'm concerned by some of the recent events regarding WineTest:
> 
> * A couple of times compilation of the official binaries broke. This got 
>   caught within a couple of days which is okayish.
> 
> * But the winehq.org upgrade completely broke test.winehq.org and it 
>   took 10 days for anyone to notice!
> 
> * Then there's the wininet:ftp, wininet:http & co breakage, also caused 
>   by the winehq.org upgrade. This time it took a whopping 24 days for 
>   anyone to notice!
>   https://www.winehq.org/pipermail/wine-devel/2019-October/151834.html
> 
> * There's also TestBot VMs that have almost completely stopped 
>   succeeding at posting WineTest results: vista, vistau64*, w8*, w864.
>   I'm probably more to blame for this than anyone else but I would have 
>   expected someone to publicly wonder what's going on with these VMs.
> 
> * Finally there's the gdi32:bitmap not-quite regression. I may have 
>   missed something but I did not see it mentioned before.
>   https://www.winehq.org/pipermail/wine-devel/2019-October/152082.html

There's for instance also the winstation tests that somehow started
failing quite recently:

https://test.winehq.org/data/tests/user32:winstation.html

without any changes to the tests:

https://source.winehq.org/git/wine.git/history/HEAD:/dlls/user32/tests/winstation.c

They (at least sometimes) succeeded at the start of the year:

https://web.archive.org/web/20190124112119/http://test.winehq.org/data/

so it must be due to some testbot change. I was actually under the
assumption that you were checking the test log after testbot changes to
see if the changes caused any failures, but I guess not.

I actually noticed the failures as soon as they started happening since
I check the test page quite often (because I get excited when I see
green stuff), but again assumed someone was monitoring that already.
Since I already look at it quite often, I can just file bug reports when
I see consistent new failures if you like.

Best,
Sven