21 Dec
2022
21 Dec
'22
11:44 p.m.
The false positives rate remained pretty low for the TestBot over the
past week, both in terms of impact to the merge requests and in the
nightly WineTest results.
Despite having a lot more test configurations, last week only about 6%
of MRs got a false positive versus 15% that got one from the GitLab CI's
single Wine test environment.
That means less annoying emails from Marvin but also that when the
TestBot now reports an issue it's really best to pay attention to it.
Here's an updated list of the test bugs causing the most false positives
in the GitLab CI:
18 mmdevapi:capture -> bug 53433
15 ntdll:threadpool -> bug 54064
11 ntdll:pipe -> bug 54078
5 ws2_32:sock -> bug 54074
5 wldap32:parse -> bug 54075
5 user32:msg -> bug 54037
5 ole32:clipboard -> bug 54005
5 mmdevapi:render -> bug 54140
And here's an update to last week's statistics:
(I attached the updated data files)
Adjusted MR False Positive rate
Week | TestBot | GitLab CI
2022-11-14 | 21.9% | 8.3%
2022-11-21 | 8.0% | 21.6%
2022-11-28 | 14.7% | 28.4%
2022-12-05 | 8.5% | 24.5%
2022-12-12 | 6.1% | 15.2%
2022-12-19 | 0.0% | 15.0% (incomplete week)
Raw MR False Positive rate
Week | TestBot | GitLab CI
2022-11-14 | 52.1% | 27.1%
2022-11-21 | 50.0% | 29.5%
2022-11-28 | 20.0% | 33.7%
2022-12-05 | 19.1% | 57.4%
2022-12-12 | 6.1% | 15.2%
2022-12-19 | 0.0% | 15.0% (incomplete week)
Test Units with "new" failures in the nightly WineTest results
Week | Total | Deduplicated
2022-11-21 | 96 | 51
2022-11-28 | 82 | 67
2022-12-05 | 108 | 53
2022-12-12 | 47 | 42
2022-12-19 | 14 | 14
--
Francois Gouget <fgouget(a)codeweavers.com>