[Bug 39512] New: Make it possible to run multiple tests in a single task

[wine-bugs at winehq.org]

https://bugs.winehq.org/show_bug.cgi?id=39512

            Bug ID: 39512
           Summary: Make it possible to run multiple tests in a single
                    task
           Product: Wine-Testbot
           Version: unspecified
          Hardware: x86
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: unknown
          Assignee: wine-bugs at winehq.org
          Reporter: fgouget at codeweavers.com
      Distribution: ---

Currently when a patch touches multiple tests the TestBot creates one task per
test per VM, and even multiple jobs if the patch touches tests for multiple
dlls.

For instance currently Wine patches are tested on four 32 bit VMs and five 64
bit VMs. So a patch that modified 2 tests in one dll and 3 in another would
create 2 jobs and 72 tasks (2 build tasks, (2+3)*(4+5) 32 bit tasks, and
(2+3)*5 64 bit tasks). The overall run time would obviously be quite long, thus
earning this type of (otherwise quite legitimate) patch the 'test bomb'
nickname.

The fix is to take the ability to build a WineTest executable with only a few
tests (see bug 39065) further so that:
 * A WineTest executable containing only the modified tests is built.
 * The TestBot then creates only one job and one task per VM to run that test
executable.
 * Just like in bug 39065 the tasks would have to specify a command line such
that only the modified tests are run. That's not just for efficiency's sake but
also to avoid getting test failures from unmodified tests.
 * Instead of having a constant 2 minutes timeout the tasks would also need to
have a timeout that's in line with the number of tests to run. This could be
something like:
   MIN(tests_count, 2) * 2 minutes + average_time * tests_count

Where average_time would be some configurable constant (probably something
around 3 to 5 seconds).

-- 
Do not reply to this email, post in Bugzilla using the
above URL to reply.
You are receiving this mail because:
You are watching all bug changes.