On Sat, 30 Mar 2019, Fabian Maurer wrote: That's precisely because the errors are not new. Normally what happens is this: * All test failures are reported on the job's page. * The TestBot compares the test's failures with those of the latest WineTest run in the same configuration. Test failures that don't appear in the WineTest run are considered "new" and are "bolded" on the job's page. For instance on the job below we see that wvistau64 has 14 new errors, while wvistau64_zh_CN has 14 pre-existing errors. https://testbot.winehq.org/JobDetails.pl?Key=50238#k301 This also proves that the TestBot may mistakenly think an error is new. There are two cases that can happen: - The error does not always happen and did not happen in the last WineTest run. I think this could be mitigated by using all the past WineTest reports and possibly keeping those for a bit more than 1 week. I think this would only represent a minor risk of false negative (you'd have to first fix a test failure and then reintroduce it). - The error contains some random text such as a memory address or some other uninitialised value. In the job above it looks like this is the case for the returned buffer size. Note that the point above also increases the chance we'll find a match in this case if there are only a few different values. * You should only get the TestBot's "I found new failures" email when there are **new** failures. This includes the case where pre-existing failures look new as above. * The same way the patches site will only mark the patch as Failed if it has **new** failures. -- Francois Gouget <fgouget(a)codeweavers.com>