TestBot changes

Greg Geldorp ggeldorp at vmware.com
Mon Apr 19 02:46:34 CDT 2010


I've made a bunch of changes to TestBot, aimed at improving its reliability
for processing wine-patches submissions and preventing false positives (where
the bot incorrectly claims a patch introduces a new failure.) For the moment
it won't send error reports directly to wine-devel, instead it sends them to
me and I'll manually forward after checking.

- Skip tests if required DLL is not available
  Previously, when running e.g. urlmon_test.exe on NT4, the test would fail
  to start if a statically imported DLL (urlmon.dll in the NT4 case) was not
  present on the system. Eventually the test would time out and this would be
  registered as a failure. The new behavior is to skip the test entirely and
  not treat it as a failure (this is the same behavior as that of
  winetest-latest.exe.)

- Add support for programs tests
  We've had cmd tests in programs/cmd/tests for a while now, patches to these
  tests were not handled properly. This should be fixed now (and obviously not
  only for programs/cmd but for any test under programs/.)

- Avoid false positives when comparing logs
  The bot tries to figure out whether new errors have been introduced by a
  patch. It does this by comparing the log generated by running the patched
  test against the log from the last winetest-latest.exe run. If there are
  new "Test failed" lines they will be flagged as new errors. When searching
  for these new lines, we have to disregard the source line numbers that are
  present in the log lines (log lines have the form "testfile.c:999: Test
  failed: Description"), since the source line numbers can easily change if
  the patch adds or removes lines near the start of the file.

- Use both author email address and max patch number for set determination
  We discussed this last week, this should be an improvement but still does
  not catch all possible scenarios.

- Run intermediate tests too
  When submitting a 4 part set, where part 2 and 3 contain changes to a test,
  we'd previously wait until all 4 parts were available, create one big patch
  file out of parts 1-4, apply that and run the tests. Now, we'll detect that
  part 2 contains a test change, apply 1-2 and run tests. Then when part 3
  becomes available we detect that it too contains a test change, apply 1-3 and
  run tests again.

More TestBot changes to come.

Ge.



More information about the wine-devel mailing list