TestBot job 50182 results: [PATCH 1/2] kernel32: Partially implement VirtualAllocExNuma

[Francois Gouget]

On Sat, 30 Mar 2019, Fabian Maurer wrote:

> This is unrelated since I didn't touch the tests, or code that the tests used.
> 
> PS: Still don't know why the PATCH is marked as OK on https://
> source.winehq.org/patches/, when the testbot says it failed.

That's precisely because the errors are not new.

Normally what happens is this:

* All test failures are reported on the job's page.

* The TestBot compares the test's failures with those of the latest 
  WineTest run in the same configuration. Test failures that don't 
  appear in the WineTest run are considered "new" and are "bolded" on 
  the job's page.
  For instance on the job below we see that wvistau64 has 14 new 
  errors, while wvistau64_zh_CN has 14 pre-existing errors.

  https://testbot.winehq.org/JobDetails.pl?Key=50238#k301

  This also proves that the TestBot may mistakenly think an error is 
  new. There are two cases that can happen:

  - The error does not always happen and did not happen in the last 
    WineTest run. I think this could be mitigated by using all the past 
    WineTest reports and possibly keeping those for a bit more than 1 
    week.
    I think this would only represent a minor risk of false negative 
    (you'd have to first fix a test failure and then reintroduce it).

  - The error contains some random text such as a memory address or 
    some other uninitialised value. In the job above it looks like this 
    is the case for the returned buffer size.
    Note that the point above also increases the chance we'll find a 
    match in this case if there are only a few different values.

* You should only get the TestBot's "I found new failures" email when 
  there are **new** failures. This includes the case where pre-existing 
  failures look new as above.

* The same way the patches site will only mark the patch as Failed if it 
  has **new** failures.


-- 
Francois Gouget <fgouget at codeweavers.com>