RFC How to get rid of "always new" TestBot false positives?

[Francois Gouget]

On Fri, 27 Mar 2020, Zebediah Figura wrote:
[...]
> (2) Remi's comment inspired another option, though it's definitely more work
> for François: lookup the line number in the relevant source file and compare
> the source line.

That's pretty hard for three reasons:
* The test reports are analyzed on the TestBot server which does not 
  have access to the Wine source (giving it access to a Wine checkout 
  is on the todo list for other reasons).
* What it needs is the patched source code whereas what it would have 
  access to is the unpatched source.
* On the server multiple processes may need access to the patched source 
  code at the same time.

Also a number of tests loop over the test cases so that they all involve 
the same lines but the failures are still distinct. For instance (from 
the set of "new" failures in a recent "32 bit WineTest: base VMs" job):

d3d11.c:2729: Test failed: Test 22: Got unexpected hr 0x8007000e (format 0x9).
d3d11.c:2729: Test failed: Test 23: Got unexpected hr 0x8007000e (format 0x9).
d3d11.c:2729: Test failed: Test 24: Got unexpected hr 0x8007000e (format 0xf).
d3d11.c:2729: Test failed: Test 25: Got unexpected hr 0x8007000e (format 0x13).

These all come from the same line but it would be wrong to say that a 
failure on iteration 22 is equal to the one on iteration 25.


-- 
Francois Gouget <fgouget at codeweavers.com>