Do we need a Mr Tests?

[Francois Gouget]

I'm concerned by some of the recent events regarding WineTest:

* A couple of times compilation of the official binaries broke. This got 
  caught within a couple of days which is okayish.

* But the winehq.org upgrade completely broke test.winehq.org and it 
  took 10 days for anyone to notice!

* Then there's the wininet:ftp, wininet:http & co breakage, also caused 
  by the winehq.org upgrade. This time it took a whopping 24 days for 
  anyone to notice!
  https://www.winehq.org/pipermail/wine-devel/2019-October/151834.html

* There's also TestBot VMs that have almost completely stopped 
  succeeding at posting WineTest results: vista, vistau64*, w8*, w864.
  I'm probably more to blame for this than anyone else but I would have 
  expected someone to publicly wonder what's going on with these VMs.

* Finally there's the gdi32:bitmap not-quite regression. I may have 
  missed something but I did not see it mentioned before.
  https://www.winehq.org/pipermail/wine-devel/2019-October/152082.html


My sinking feeling is:
* Nobody looks at test.winehq.org.
* Major WineTest regressions can sail through without anyone noticing.

My feeling is that nobody feels responsible for making sure 
test.winehq.org works and for spotting regressions.

Hence why I think we may need someone whose responsibility would be to 
monitor test.winehq.org, analyze the WineTest results, diagnose new 
issues and report them.


I did something like that a few times but it takes quite a bit of time 
so I had to stop. Essentially it went something like this:

* Every couple of weeks, open the results for the latest build:
  https://test.winehq.org/data/7d954f23356f0aaf49b1ef0c4bed83041cb41c08/

* For each line that has a failure on Windows, open the corresponding 
  test page. For instance:
  https://test.winehq.org/data/tests/advapi32:service.html

* If the tests start failing consistenly on a specific date on at least 
  one platform, dig in to figure out why.
  For instance:
  https://test.winehq.org/data/tests/gdi32:bitmap.html
  https://test.winehq.org/data/tests/wininet:http.html

* Likewise, if a test has been consistently getting 13 failures on a 
  platform and now consistently gets 19 it may be because a commit 
  introduced new failures.
  For instance:
  https://test.winehq.org/data/tests/advapi32:service.html

This requires regular checks otherwise failure is the new normal and 
there is nothing to see.



-- 
Francois Gouget <fgouget at free.fr>              http://fgouget.free.fr/
                  A black hole is just God dividing by zero.