RFC How to get rid of "always new" TestBot false positives?

[Francois Gouget]

On Fri, 27 Mar 2020, Jefferson Carpenter wrote:
[...]
> >      Test failed: hwnd ((0028050C)) message 0738
> 
> For this case in particular, I don't believe it's useful to output the hwnd at
> all, unless that's a useful piece of information in diagnosing the test
> failure.

I prefer removing useless information from the failure messages too but 
it's not always easy to determine what's useful.

Here the test traces some window handles (parent and child 1 to 4) so 
having the hwnd of the window the unexpected message is sent to can 
allow matching it to one of those... or not, which I guess is useful.

Maybe not specifically in the above message but at least in the "wrong 
parent/child" failure messages below.


> >      Test failed: WaitCommEvent used ((1594)) ms for waiting
> >      Test failed: Unexpected date value ((24/03/2020)), got ((2020/03/24))
> > [3]
> 
> I agree that the delimiter must be chosen carefully.  I would prefer something
> like TESTBOT_IGNORE(...) which would never show up in a failure message
> otherwise, and would even have some possibility of being discovered by
> grepping the source code.

That's a good point. "TESTBOT_IGNORE" is relatively clear but it's also 
really quite verbose:

    ok(!IsChild(desktop, parent), "wrong parent/child %p/%p\n", desktop, parent);
->  ok(!IsChild(desktop, parent), "wrong parent/child TESTBOT_IGNORE(%p/%p)\n", desktop, parent);

   ok(!strcmp(buff, time), "Expected %s, got %s\n", time, buff);
-> ok(!strcmp(buff, time), "Expected TESTBOT_IGNORE(%s), got TESTBOT_IGNORE(%s)\n", time, buff);

None of '((', '<<', '[[' or '{{' is very greppable because of uses in C, 
js and the tests. '<[' is somewhat common in XML files, so maybe a mixed 
form like '<{' would be the best option: git grep '<[{]' */*/tests/* 
returns nothing.

So we'd have:

   ok(!IsChild(desktop, parent), "wrong parent/child <{%p/%p}>\n", desktop, parent);
   ok(!strcmp(buff, time), "Expected <{%s}>, got <{%s}>\n", time, buff);



-- 
Francois Gouget <fgouget at codeweavers.com>