TestBot changes
Paul Vriens
paul.vriens.wine at gmail.com
Mon Apr 19 03:05:50 CDT 2010
On 04/19/2010 09:46 AM, Greg Geldorp wrote:
> I've made a bunch of changes to TestBot, aimed at improving its reliability
> for processing wine-patches submissions and preventing false positives (where
> the bot incorrectly claims a patch introduces a new failure.) For the moment
> it won't send error reports directly to wine-devel, instead it sends them to
> me and I'll manually forward after checking.
>
> - Skip tests if required DLL is not available
> Previously, when running e.g. urlmon_test.exe on NT4, the test would fail
> to start if a statically imported DLL (urlmon.dll in the NT4 case) was not
> present on the system. Eventually the test would time out and this would be
> registered as a failure. The new behavior is to skip the test entirely and
> not treat it as a failure (this is the same behavior as that of
> winetest-latest.exe.)
Winetest.exe has some extra testing for com and .net provided dll's.
Does the testbot take this into account as well?
>
> - Add support for programs tests
> We've had cmd tests in programs/cmd/tests for a while now, patches to these
> tests were not handled properly. This should be fixed now (and obviously not
> only for programs/cmd but for any test under programs/.)
>
> - Avoid false positives when comparing logs
> The bot tries to figure out whether new errors have been introduced by a
> patch. It does this by comparing the log generated by running the patched
> test against the log from the last winetest-latest.exe run. If there are
> new "Test failed" lines they will be flagged as new errors. When searching
> for these new lines, we have to disregard the source line numbers that are
> present in the log lines (log lines have the form "testfile.c:999: Test
> failed: Description"), since the source line numbers can easily change if
> the patch adds or removes lines near the start of the file.
>
> - Use both author email address and max patch number for set determination
> We discussed this last week, this should be an improvement but still does
> not catch all possible scenarios.
>
> - Run intermediate tests too
> When submitting a 4 part set, where part 2 and 3 contain changes to a test,
> we'd previously wait until all 4 parts were available, create one big patch
> file out of parts 1-4, apply that and run the tests. Now, we'll detect that
> part 2 contains a test change, apply 1-2 and run tests. Then when part 3
> becomes available we detect that it too contains a test change, apply 1-3 and
> run tests again.
>
> More TestBot changes to come.
Your work is greatly appreciated. It already cuts down on my job as I
don't have to chase people that much ;)
--
Cheers,
Paul.
More information about the wine-devel
mailing list