RFC How to get rid of "always new" TestBot false positives?
jeffersoncarpenter2 at gmail.com
Fri Mar 27 07:25:35 CDT 2020
On 3/27/2020 11:53 AM, Francois Gouget wrote:
> The issue is that some test failures include values that change on every
> run which prevents the TestBot from knowing whether they are new or not.
> In turn this means the TestBot systematically blames the author of
> whichever patch is being tested for these failures.
So you're wanting to add a sort of comment syntax so that TestBot can
ignore parts of failure messages referring to transient data.
> For instance one could enclose the irrelevant parts in
> double-parentheses  as follows:
> Test failed: hwnd ((0028050C)) message 0738
For this case in particular, I don't believe it's useful to output the
hwnd at all, unless that's a useful piece of information in diagnosing
the test failure.
> Test failed: WaitCommEvent used ((1594)) ms for waiting
> Test failed: Unexpected date value ((24/03/2020)), got ((2020/03/24)) 
I agree that the delimiter must be chosen carefully. I would prefer
something like TESTBOT_IGNORE(...) which would never show up in a
failure message otherwise, and would even have some possibility of being
discovered by grepping the source code.
More information about the wine-devel