GSoC Idea: Winetest Graphical User Interface

[Francois Gouget]

On Sun, 13 Mar 2016, Zhenbo Li wrote:

> Hello,
> 
> I'm Zhenbo Li. This year, I wish to work on a project related to wine
> test suite.
> 
> Previously, I sent a mail for Winetest Scripting Interface.
> 
> A better winetest.exe should have such functions:
> 
> 1. Choose which dlls to be tested. Users could use GUI, CLI, or a config file.

Being able to run a subset of the tests (e.g. kernel32:actctx to 
ole32:usrmarshal) could be useful from time to time to figure out which 
test crashes a VM or breaks one of the following tests. To be useful one 
would have to be able to easily pick an arbitrary range of tests, for 
instance by selecting all of them in a list.


> 2. Show the result in winetest, or render it to html

This does not seem useful. It would be nice to be able to both send the 
result and keep it in a local log but to do that I have just taken to 
run winetest twice: first storing the result in a file without 
submitting the result, then just to send that file.

Much more useful would be to know why test.winehq.org rejects a test 
result because that can be a real mystery sometimes. I guess getting 
immediate feedback in WineTest could be useful sometimes, but for my use 
case it would be much better to get that information from the website 
after the fact since the VM that ran the tests was likely reverted when 
I get around to check what happened.

I'm thinking one could have a per commit page listing the rejected 
submissions, one per line with something like:

2016/03/13 01:02:03 fg-sol1111   Too many bad test results (45 failures, 4 skipped, 2 missing dlls)
2016/03/13 01:12:33 fg-winxp-1sp File too big (1602KB)
...


> 3. Allow to leave tag and email blank if the user doesn't want to
> submit the result to test.winehq.org, and they can fill them when
> submitting them.


> I'm wondering whether this job would be enough for a summer, and I'm
> glad to write a script for wine test suite as another part of my job.

It does not feel like it would be sufficient.


Aaryaman Vasishta wrote:
> To be more specific, if you have multiple test functions A(), B() and 
> C() inside a test .c file, there could be a GUI to select/deselect 
> which tests to submit (say, A() and B()) to the testbot and then 
> generate the diff and submit it via a POST request or something.

WineTest does not have the Wine source and the functions are just that: 
functions. There is no way to call just one of them. That would also be 
much more fine grained than something like shell32:shlexec and the 
individual functions may need some set up to be done before hand so 
calling them directly may not make sense.

But if you know you're interested in a specific test function that 
means you've looked at the source, presumably in an editor. So why not 
generate the diff there and submit it to the TestBot? Firing off 
WineTest seems like a more complex and roundabout way of going about it.

-- 
Francois Gouget <fgouget at free.fr>              http://fgouget.free.fr/
    Indifference will certainly be the downfall of mankind, but who cares?