[Bug 48653] Automate checking the WTBS jobs

[WineHQ Bugzilla]

https://bugs.winehq.org/show_bug.cgi?id=48653

--- Comment #1 from François Gouget <fgouget at codeweavers.com> ---
An alternative to the above property system would to be embed a .errors file in
the commit message.

.errors can both store properties like the ones described above, and groups of
errors. Those can be compared to the task's new errors, i.e. those generated by
the WTBS patch, to ensure we got the expected result. Note that the comparison
already takes care of ignoring changes like line numbers and variable parts
like crash addresses, etc.


For instance:
--- WTBS.Results ---
p WTBS.Job.Remarks WTBS Unreported test failures (kernel32:comm).
p WTBS.Job.Status completed
p WTBS.win.Status completed
p WTBS.win.TestUnits kernel32:comm
p WTBS.wine.TestUnits kernel32:comm
g 0 kernel32
n 0 comm.c:2210: Test failed: WTBS Simulate an unreported test failure
g 0 Report validation errors
n 0 kernel32:comm has unaccounted for failure messages
n 0 kernel32:comm returned success despite having failures
--- WTBS.Results ---

This scheme would still need some adjustments to allow for different results on
Windows and Wine VMs (prefix the group names?). It may also prove hard to
maintain if the TestBot error messages change too often.

Note: The WTBS convention is to put the test units being used for the test in
parentheses at the end of the subject line. This may be leveraged to provide a
good default for the TestUnits properties (the exception being test=module
again).

-- 
Do not reply to this email, post in Bugzilla using the
above URL to reply.
You are receiving this mail because:
You are watching all bug changes.