GSoC Idea: Tools - Winetest Scripting Interface

[Zhenbo Li]

Hello, I'm Zhenbo Li. I participated in GSoC twice. This year, I want
to work on "Tools - Winetest Scripting Interface". After "make test",
checking the output in stdout manually is horrible. As ok() macro is
used quite widely, my patches should cause no surprise. That is to
say, other developers need to read neither my patches nor the
changelog to continue their jobs, although ok() macro and other tools
for test have been changed.

I think there will be two steps for the goal:

1. ok() macro
=====================

int winetest_vok( int condition, const char *msg, __winetest_va_list args )
{
    tls_data* data=get_tls_data();
    if (data->todo_level){...}
    else
    {
        if (!condition)
        {
            printf( "%s:%d: Test failed: ",
                    data->current_file, data->current_line );
            vprintf(msg, args);
            InterlockedIncrement(&failures);
            return 0;
        }
        else{.....}
    }
}

Instead of printing all the data to stderr, I think we can write a wrapper
static void test_print(const char *s)
{
    if(want_print_to_file)
    fprintf(file_handle, s);
    printf(s);
}

It needs some work to make sure that the file_handle is opened/closed
successfully. With that, we can record the failed tests to a file, and
developers don't need to find them in stdout anymore.

2. compare the results
=======================
On the idea page, it says that any scripting language is acceptable. I
prefer Python3. To make the code simple, I think editing runtest
script is the best solution.

Currently, our logic in Makefile is

htmllocation.ok:
    ../../../tools/runtest $(RUNTESTFLAGS) -T ../../.. -M mshtml.dll
-p mshtml_test.exe.so htmllocation && touch $@

And in runtest, we have
exec $WINETEST_WRAPPER "$topobjdir/wine" "$program" "$@"

I don't want to touch Makefile. Instead, I'm going to add some options
for runtest. For example, one can use
../../../tools/runtest $(RUNTESTFLAGS) -new_option
expected_errors_generated before -T ../../.. -M mshtml.dll -p
mshtml_test.exe.so htmllocation

And in runtest script, we can expected_errors_generated and
new_errors_got, then tell that whether the test result has been
changed. It should be easy for developers to write a script for git
bisect or other circumstance.

Problems
=========
This approach seems to be quite simple, and there are some conditions
can't be handled. For example

void test_foo(sturct Data *d){
    ret = foo(d);
    ok(ret, "foo(%s) is %d\n", data_to_str(d), ret);
}
struct Data list[]={a,b,c,d,e};
void test(){
    for(i in list)
        test_foo(i);
}

I think we should not do too much "smart deduce" here, and don't hide
what's happening to other developers. Maybe we can add another option
for runtest script: --scrict=false

This is my humble idea for Wine's GSoC 2016. If this project goes
well, working on wine test suit will be much more satisfying(although
still boring). I appreciate for your comment.

With you a happy weekend.

-- 
Have a nice day!
Zhenbo Li