TestBot changes

[Paul Vriens]

On 04/19/2010 09:46 AM, Greg Geldorp wrote:
> I've made a bunch of changes to TestBot, aimed at improving its reliability
> for processing wine-patches submissions and preventing false positives (where
> the bot incorrectly claims a patch introduces a new failure.) For the moment
> it won't send error reports directly to wine-devel, instead it sends them to
> me and I'll manually forward after checking.
>
> - Skip tests if required DLL is not available
>    Previously, when running e.g. urlmon_test.exe on NT4, the test would fail
>    to start if a statically imported DLL (urlmon.dll in the NT4 case) was not
>    present on the system. Eventually the test would time out and this would be
>    registered as a failure. The new behavior is to skip the test entirely and
>    not treat it as a failure (this is the same behavior as that of
>    winetest-latest.exe.)

Winetest.exe has some extra testing for com and .net provided dll's. 
Does the testbot take this into account as well?

>
> - Add support for programs tests
>    We've had cmd tests in programs/cmd/tests for a while now, patches to these
>    tests were not handled properly. This should be fixed now (and obviously not
>    only for programs/cmd but for any test under programs/.)
>
> - Avoid false positives when comparing logs
>    The bot tries to figure out whether new errors have been introduced by a
>    patch. It does this by comparing the log generated by running the patched
>    test against the log from the last winetest-latest.exe run. If there are
>    new "Test failed" lines they will be flagged as new errors. When searching
>    for these new lines, we have to disregard the source line numbers that are
>    present in the log lines (log lines have the form "testfile.c:999: Test
>    failed: Description"), since the source line numbers can easily change if
>    the patch adds or removes lines near the start of the file.
>
> - Use both author email address and max patch number for set determination
>    We discussed this last week, this should be an improvement but still does
>    not catch all possible scenarios.
>
> - Run intermediate tests too
>    When submitting a 4 part set, where part 2 and 3 contain changes to a test,
>    we'd previously wait until all 4 parts were available, create one big patch
>    file out of parts 1-4, apply that and run the tests. Now, we'll detect that
>    part 2 contains a test change, apply 1-2 and run tests. Then when part 3
>    becomes available we detect that it too contains a test change, apply 1-3 and
>    run tests again.
>
> More TestBot changes to come.

Your work is greatly appreciated. It already cuts down on my job as I 
don't have to chase people that much ;)

-- 
Cheers,

Paul.