Troublesome test units: grep -h -v -P '^(#| +[0-9])' failures-mr.txt | sed -n '/week 12-02$/,/week 11-28$/p' | sed -e 's/[^a-z0-9:._]/\n/g' | egrep '[a-z0-9]:[a-z0-9]' | sort | uniq -c | sort -n -r # or to filter on days instead of weeks sed -n '/^12-05/,/^12-02/p' | # or to exclude already diagnosed / filtered out issues sed -e 's/[a-z][a-z0-9:._]*:[a-z][a-z0-9:._]* ->//g' | # or to exclude the GitLab CI failures sed -e 's/([^)]*)//g' | # or to keep on the GitLab CI failures sed -e 's/^.*[(]\([^)]*\)[)].*$/\1/' -e t -e d | TestBot false positive rate and more: for t in "[.]" "N" "F" "B" "D" "O" "E" "S" "[?]" "."; do count=`sed -n '/week 12-02$/,/week 11-28$/p' failures-mr.txt | egrep -i "^$t. [0-9][0-9][0-9]" | wc -l`; [ $count -ne 0 ] && echo " $t -> $count"; done GitLab CI false positive rate and more: for t in "[.]" "N" "F" "B" "D" "O" "E" "S" "[?]" "."; do count=`sed -n '/week 12-02$/,/week 11-28$/p' failures-mr.txt | egrep -i "^.$t [0-9][0-9][0-9]" | wc -l`; [ $count -ne 0 ] && echo " $t -> $count"; done For importing in spreadsheets: (p=13;for d in 12 09 08 07 06 05 02 01;do echo -n "2022/12/$p,"; for t in "[.]" "N" "F" "B" "D" "O" "E" "S" "[?]" "."; do count=`sed -n "/^12-$p/,/^12-$d/p" failures-mr.txt | egrep -i "^$t. [0-9][0-9][0-9]" | wc -l`; echo -n "$count,"; done; p=$d; echo; done; p=12-01;for d in 30 29 28 26 24 23 22 21 18 17 16 15 14 11;do echo -n "2022/$p,"; for t in "[.]" "N" "F" "B" "D" "O" "E" "S" "[?]" "."; do count=`sed -n "/^$p/,/^11-$d/p" failures-mr.txt | egrep -i "^$t. [0-9][0-9][0-9]" | wc -l`; echo -n "$count,"; done; p=11-$d; echo; done) # One character classification codes: # . Success # The CI found no failure attributable to the MR. # N False negative # An MR that got merged and broke a test with no advance warning of the CI. # F False positive # Identifies cases where the CI system incorrectly claimed the MR introduces # new failures. This is typically the case when the failures that are already # present in nightly WineTest results. # B Bad merge # Identifies MRs that break a test and got merged anyway. Whether an MR is # F or B is mostly determined after the fact. # D Collateral Damage from a bad merge # This identifies the false positives (aka collateral damage) caused by one # of the bad merges above. # O Massive outside interference # This is not an intermittent false positive intrinsic to the test, but a # failure resulting from a change outside the Wine infrastructure, such as # when the configuration of a server used for the tests changes and breaks the # tests that depend on it. # E BotError # And internal CI bug. Mostly relevant for TestBot results. # S Patch series mixup # Cases where it looks like the TestBot mixed up two patch series, resulting # in a patch that cannot be applied and thus no actual test results. # ? Status to be determined # # A merge request may have more than one type of failure. In that case the # priority order is: B N F D O E S ? 12-16 --- week 12-12 12-13 .f 1750 (winhttp:notification) .f 1802 (ntdll:pipe -> bug 54078) .. 1803 .. 1808 .. 1776 .f 1804 (wmvcore:wmvcore) 12-12 .. 1793 .. 1796 .. 1766 .. 1792 .. 1768 .. 1772 .. 1763 .. 1791 .. 1789 12-09 --- week 12-02 .o 1751 (winhttp:notification -> bug 54098) .f 1681 (mmdevapi:capture user32:msg) (winhttp:notification -> bug 54098) .o 1740 (winhttp:notification -> bug 54098) .f 1757 (build:windows-on-unix) .f 1761 (ole32:clipboard -> bug 54005) .f 1758 (ntdll:pipe -> bug 54078) (ws2_32:sock -> bug 54074 + bug 52474) .. 1759 .f 1631 (ntdll:pipe -> bug 54078) (wmvcore:wmvcore) .. 1638 oo 1721 winhttp:notification -> bug 54098 (winhttp:notification -> bug 54098) .f 1754 (build:windows-on-unix) .. 1760 .. 1582 12-08 .o 1727 (winhttp:notification -> bug 54098) .o 1739 (winhttp:notification -> bug 54098) .o 1738 (winhttp:notification -> bug 54098) .o 1737 (winhttp:notification -> bug 54098) fo 1635 ws2_32:afd -> new bug 54113 (winhttp:notification -> bug 54098) .o 1748 (winhttp:notification -> bug 54098) .f 1747 (ntdll:threadpool -> bug 54064) (winhttp:notification -> bug 54098) .o 1745 (winhttp:notification -> bug 54098) .o 906 (winhttp:notification -> bug 54098) .o 1744 (winhttp:notification -> bug 54098) .f 1742 (winhttp:notification -> bug 54098) (ws2_32:afd -> new bug 54113) .f 1736 taskschd:scheduler -> new bug 54109 (winhttp:notification -> bug 54098) .f 1701 (ntdll:threadpool -> bug 54064) (winhttp:notification -> bug 54098) .o 1743 (winhttp:notification -> bug 54098) .o 1734 (winhttp:notification -> bug 54098) .o 1735 (winhttp:notification -> bug 54098) .. 1741 12-07 Bo 1732 taskschd:scheduler -> new bug 54105 + new bug 54106 (winhttp:notification -> bug 54098) .o 1666 (ntdll:threadpool -> bug 54064) (winhttp:notification -> bug 54098) ff 1705 ws2_32:sock -> bug 54073 variant (ddraw:ddraw7) (ntdll:pipe -> bug 54078) (winhttp:notification -> bug 54098) .o 1726 (winhttp:notification -> bug 54098) .o 1730 (rpcrt4:server) (winhttp:notification -> bug 54098) .o 1729 (ntdll:threadpool -> bug 54064) (winhttp:notification -> bug 54098) .o 1728 (ntdll:threadpool -> bug 54064) (winhttp:notification -> bug 54098) Fo 1731 msctf:inputprocessor (winhttp:notification -> bug 54098) oo 1713 winhttp:notification -> bug 54098 (winhttp:notification -> bug 54098) So 1717 ??? (winhttp:notification -> bug 54098) .o 1719 (winhttp:notification -> bug 54098) oo 1718 winhttp:notification -> bug 54098 (winhttp:notification -> bug 54098) oo 1720 winhttp:notification -> bug 54098 (winhttp:notification -> bug 54098) of 1722 winhttp:notification -> bug 54098 (ntdll:threadpool -> bug 54064) (winhttp:notification -> bug 54098) oo 1723 winhttp:notification -> bug 54098 (winhttp:notification -> bug 54098) oo 1716 winhttp:notification -> bug 54098 (winhttp:notification -> bug 54098) .o 1698 (winhttp:notification -> bug 54098) .f 1715 (winhttp:notification -> bug 54098) (ws2_32:sock -> bug 54074) oo 1724 winhttp:notification -> bug 54098 (winhttp:notification -> bug 54098) 12-06 .. 1712 .. 1704 .. 1702 .. 1692 .. 1693 .. 1618 f. 1708 wmvcore:wmvcore => unique, ignore ff 1703 quartz:filtermapper -> new bug 54095 (mshtml:misc user32:menu) .. 1709 .. 1711 .f 1700 (ntdll:threadpool -> bug 54064) .f 1688 (ntdll:pipe -> bug 54078) (user32:clipboard) .. 1697 .f 1699 (urlmon:protocol) (ws2_32:sock -> bug 54074) .. 1696 .. 1694 .. 1695 f. 1691 kernel32:process -> new bug 54094 .. 1690 .. 1689 .. 1181 .. 1687 12-05 .f 1630 (ntdll:threadpool -> bug 54064) .. 1664 .. 1602 .. 1677 .. 1529 .. 1660 .f 1685 (ntdll:threadpool -> bug 54064) .f 1668 (mmdevapi:capture -> bug 53433) (qasf:asfreader) .. 1676 f. 1675 shell32:autocomplete -> new bug 54090 .. 1678 .. 1679 .. 1646 .. 1643 .. 1680 .. 1673 .f 1667 (ws2_32:afd -> new bug 54113) (ws2_32:sock -> bug 52474) .f 1669 (wininet:http -> bug 54084) (ws2_32:sock -> bug 54073) f. 1547 mmdevapi:capture -> bug 53265 variant .. 1663 .f 1672 (dxgi:dxgi explorer.exe:explorer ntdll:file) .. 1671 .. 1670 12-02 --- week 11-28 .f 1622 (user32:menu) (user32:msg -> bug 54037) .. 1542 .. 1654 .f 1659 (wmvcore:wmvcore -> bug 54042) .. 1623 .. 1653 .. 1640 .f 1647 (mmdevapi:capture -> bug 53433) .. 1644 .. 1645 .. 1649 .. 1651 .. 1629 .. 1655 .f 1656 (ole32:clipboard -> bug 54005) .. 1657 .f 1658 (mmdevapi:capture -> bug 53433) (ole32:clipboard -> bug 54005) .. 1648 .. 1650 .. 1652 .f 1617 (ntdll:pipe -> new bug 54078) (user32:msg -> bug 54037) 12-01 .. 1642 .. 1641 .. 1639 .. 1632 .f 1523 (timeout) .. 1634 .. 1606 .. 1636 f. 1619 user32:input -> new bug 54053 + bug 51477 .f 1502 (mmdevapi:capture -> bug 53433) Ff 1627 mshtml:misc (ntdll:pipe -> new bug 54078) .. 1624 .. 1510 .f 1608 (mmdevapi:render -> new bug 54140) (user32:msg -> bug 54037) .f 1615 (ntdll:threadpool -> new bug 54064) .. 1616 .. 1621 .. 1620 .. 1569 .o 1579 (wldap32:parse -> new bug 54075) .. 1571 11-30 f. 1560 winhttp:notification -> new bug 54052 ff 1598 shell32:shelllink -> new bug 54061 (mmdevapi:capture -> bug 53433) (mmdevapi:render -> new bug 54140) (ntdll:pipe -> new bug 54078) .. 1503 .. 1614 .. 1611 .. 1613 .. 1593 .. 1599 .. 1601 df 1454 MR!1399 fallout (ntdll:threadpool -> new bug 54064) f. 1553 winhttp:notification -> new bug 54052 e. 1604 The previous 1 run(s) ... .. 1556 .f 1607 (mmdevapi:capture -> bug 53433) .. 1609 ff 1199 wmvcore:wmvcore -> new bug 54039 (mmdevapi:render -> new bug 54140) .. 1520 .f 1605 (mmdevapi:capture -> bug 53433) .f 1603 (mmdevapi:capture -> bug 53433) .. 1567 .f 1564 (mmdevapi:render -> new bug 54140) 11-29 .. 1555 ff 1575 shell32:shelllink -> new bug 54061 advapi32:security -> bug 54014 (mmdevapi:capture -> bug 53433) oo 1581 wldap32:parse -> new bug 54075 (wldap32:parse -> new bug 54075) oo 1574 wldap32:parse -> new bug 54075 shell32:autocomplete -> new bug 54090 (wldap32:parse -> new bug 54075) .. 1522 .. 1539 .. 1551 .. 1597 .f 1326 (ntdll:threadpool -> new bug 54064) (user32:msg -> new bug 54036) (user32:win -> gitlab-specific) .f 1596 (user32:msg -> new bug 54037) ff 1595 advapi32:security -> bug 54014 (ntdll:pipe -> new bug 54078) fo 1585 msctf:inputprocessor -> new bug 54062 wldap32:parse -> new bug 54075 (wldap32:parse -> new bug 54075) fo 1583 advapi32:security -> bug 54014 wldap32:parse -> new bug 54075 wmvcore:wmvcore -> new bug 54039 ws2_32:sock -> new bug 54073 (wldap32:parse -> new bug 54075) .. 1566 f. 1594 advapi32:security -> bug 54014 .. 1592 .. 1577 11-28 .. 1562 ff 1561 vbscript:run -> new bug 54003 (ddraw:ddraw2) (ole32:clipboard -> new bug 54005) (ws2_32:sock -> new bug 54074) .. 1572 .. 1548 .f 1576 (mmdevapi:capture -> bug 53433) .f 1573 (mmdevapi:capture -> bug 53433) f. 1549 d3d9:d3d9ex -> new bug 53979 ntoskrnl.exe:driver_pnp -> new bug 54004 .. 1558 .. 1563 ef 1526 The previous 1 run(s) ... (ole32:clipboard -> new bug 54005) .. 1568 .. 1565 .. 1570 f. 1557 winhttp:notification -> new bug 54052 .f 1536 (ntdll:file) 11-26 --- week 11-21 TestBot [E] -> 2 [S] -> 5 [?] -> 3 [F] -> 33 including 29 MR!1399 fallout [B] -> 1 [N] -> 0 [.] -> 44 . -> 88 GitLab CI [c] -> 26 [.] -> 62 . -> 88 12 kernel32:process 4 ws2_32:sock 4 mmdevapi:capture 3 ntdll:threadpool 2 vbscript:run 2 ntdll:pipe 2 kernel32:heap 1 user32:msg 1 user32:menu 1 urlmon:protocol 1 uiautomationcore:uiautomation 1 ucrtbase:cpp 1 shell32:shelllink 1 ntdll:file 1 mshtml:misc 1 mmdevapi:render 1 d3drm:d3drm 1 d3d9:d3d9ex 1 advapi32:security .f 1546 (ws2_32:sock -> new bug 54073) f. 1550 uiautomationcore:uiautomation => unique, ignore .f 1519 (user32:msg -> new bug 54037) .. 1544 ff 1537 advapi32:security -> bug 54014 (ntdll:threadpool -> new bug 54064) .. 1535 .. 1540 .. 1541 11-24 d. 1489 MR!1399 fallout .f 1528 (mmdevapi:render -> new bug 54140) .. 1422 f. 1534 vbscript:run -> new bug 54003 f. 1538 d3d9:d3d9ex -> new bug 53979 .f 1527 (kernel32:process) .. 1530 .f 1531 (mmdevapi:capture -> bug 53433) .. 1532 d. 1427 MR!1399 fallout 11-23 kernel32:process -> c#b00a831d fixed since dd 1297 MR!1399 fallout (kernel32:process) .d 1524 (kernel32:process -> fixed) (ws2_32:sock -> new bug 54073) .f 1525 (kernel32:process -> fixed) (ntdll:pipe -> new bug 54078) (ws2_32:sock -> new bug 54073) .f 1504 (kernel32:process -> fixed) (ntdll:pipe -> new bug 54078) .d 1514 (kernel32:process -> fixed) .d 1513 (kernel32:process -> fixed) .d 1511 (kernel32:process -> fixed) .d 1521 (kernel32:process -> fixed) .d 1515 (kernel32:process -> fixed) ef 1506 The previous 1 run(s) ... (build-linux timeout) .. 1500 d. 1456 MR!1399 fallout Sf 1512 ??? (kernel32:process -> fixed) (mshtml:misc user32:menu) d. 1423 MR!1399 fallout .. 1384 d. 1491 MR!1399 fallout .f 1509 (mmdevapi:capture -> bug 53433) (urlmon:protocol -> new bug 54080) .. 1507 .f 1508 (ntdll:file) (ntdll:threadpool -> new bug 54064) .. 1505 .f 1485 (ws2_32:sock -> new bug 54074) ef 1447 The previous 1 run(s) ... (build-linux timeout) 11-22 d. 1461 MR!1399 fallout .. 1453 f. 1470 kernel32:heap => unique, ignore .f 1499 (kernel32:heap => unique, ignore) .. 1497 .. 1501 d. 1442 MR!1399 fallout Bf 1495 vbscript:run -> fixed since (shell32:shelllink -> new bug 54061) .. 1496 d. 1490 MR!1399 fallout .. 1493 d. 1474 MR!1399 fallout d. 1476 MR!1399 fallout .. 1482 d. 1464 MR!1399 fallout .. 1486 .. 1487 d. 1460 MR!1399 fallout d. 1466 MR!1399 fallout .. 1479 .. 1481 df 1483 MR!1399 fallout (mmdevapi:capture -> bug 53433) .. 1484 .. 1480 d. 1488 MR!1399 fallout S. 1478 ??? S. 1448 ??? S. 1477 ??? 11-21 f. 1473 ucrtbase:cpp => unique, ignore .. 1475 S. 1459 ??? d. 1435 MR!1399 fallout df 1462 MR!1399 fallout (mmdevapi:capture -> bug 53433) d. 1467 MR!1399 fallout d. 1451 MR!1399 fallout df 1450 MR!1399 fallout (ntdll:threadpool -> new bug 54064) d. 1452 MR!1399 fallout .. 1458 .. 1463 f. 1457 d3drm:d3drm -> new bug 53974 d. 1472 MR!1399 fallout d. 1449 MR!1399 fallout d. 1343 MR!1399 fallout .. 1416 d. 1471 MR!1399 fallout d. 1465 MR!1399 fallout d. 1468 MR!1399 fallout d. 1469 MR!1399 fallout 11-18 --- week 11-14 TestBot [E] -> 3 [S] -> 0 [?] -> 1 [F] -> 43 including 10 MR!1399 fallout [B] -> 2 [N] -> 1 [.] -> 46 . -> 96 GitLab CI [c] -> 26 [.] -> 70 . -> 96 38 crypt32:cert 6 d3drm:d3drm 4 advapi32:security 3 winediag:d3d_device_create 3 mmdevapi:capture 3 d3d8:device 2 wmvcore:wmvcore 2 user32:win 2 user32:msg 2 ole32:clipboard 2 httpapi:httpapi 2 d3d9:device 1 shell32:shelllink 1 quartz:systemclock 1 mshtml:htmldoc 1 ntoskrnl.exe:ntoskrnl 1 dinput:hid 1 d3d11:d3d11 d. 1334 MR!1399 fallout d. 1434 MR!1399 fallout d. 1426 MR!1399 fallout ?. 1436 MR!1399 fallout + bad patch??? d. 1373 mshtml:htmldoc => only in this MR, ignore d. 1440 MR!1399 fallout df 1441 MR!1399 fallout (shell32:shelllink -> new bug 54061) d. 1438 MR!1399 fallout d. 1417 MR!1399 fallout d. 1433 MR!1399 fallout .. 1431 .. 1430 .. 1429 .. 1428 eo 1231 The previous 1 run(s) ... (crypt32:cert -> run on 2022-11-18 but fixed by MR!1360 on 2022-11-14!) d. 1439 MR!1399 fallout ff 1413 winediag:d3d_device_create -> lumped with bug 53975 (mmdevapi:capture -> bug 53433) .. 914 .. 1392 oo 966 crypt32:cert -> run on 2022-11-14, fixed by MR!1360 (crypt32:cert -> run on 2022-11-14, fixed by MR!1360) .. 1424 N. 1432 good but -> bug 53977 11-17 f. 1419 d3drm:d3drm -> new bug 53975 .. 1420 .. 1340 B. 1398 ntoskrnl.exe:ntoskrnl -> bug 53976 .. 1412 f. 1400 dinput:hid -> bug 53525 fixed context regular expression .. 1409 e. 1394 The previous 1 run(s) ... .. 1415 .. 1335 .. 1396 f. 1397 httpapi:httpapi -> new bug 53980 .. 1403 B. 1399 broke winetest -> bug 53963 .f 1405 (build-linux cc error) .. 1401 .. 1406 f. 1404 d3drm:d3drm -> new bug 53974 .f 1402 (mmdevapi:capture -> bug 53433) .. 1410 .. 1407 f. 1411 d3drm:d3drm -> new bug 53974 .. 1408 11-16 .. 1389 .. 1374 f. 1391 d3d8:device -> bug 52983 variant .. 1390 .. 1386 .o 1295 (crypt32:cert -> run on 2022-11-16 but fixed by MR!1360 on 2022-11-14!) .. 1365 f. 1385 d3drm:d3drm -> new bug 53974 d3d9:device -> bug 52983 .. 1388 .. 1387 f. 1382 ole32:clipboard -> new bug 54005 .o 1351 (crypt32:cert -> run on 2022-11-16 but fixed by MR!1360 on 2022-11-14!) .. 1381 .. 1380 .. 1376 f. 1379 advapi32:security -> bug 54014 .. 1378 .. 1375 .. 1383 11-15 f. 1344 d3d11:d3d11 -> bug 53252 ef 1370 The previous 1 run(s) ... (quartz:systemclock) .. 1369 f. 1372 d3drm:d3drm -> bug 53975 fo 1350 crypt32:cert -> fixed by MR!1360 wmvcore:wmvcore -> new bug 54041 (crypt32:cert -> fixed by MR!1360) oo 1366 crypt32:cert -> fixed by MR!1360 (crypt32:cert -> fixed by MR!1360) .. 1368 oo 1361 crypt32:cert -> fixed by MR!1360 (crypt32:cert -> fixed by MR!1360) f. 1298 crypt32:cert -> fixed by MR!1360 d3d8:device -> bug 52983 variant d3d9:device -> bug 52983 oo 1367 crypt32:cert -> fixed by MR!1360 (crypt32:cert -> fixed by MR!1360) 11-14 fo 1363 crypt32:cert -> fixed by MR!1360 advapi32:security -> bug 54014 (crypt32:cert -> fixed by MR!1360) oo 1317 crypt32:cert -> fixed by MR!1360 (crypt32:cert -> fixed by MR!1360) .f 1362 (crypt32:cert -> fixed by MR!1360) (mmdevapi:capture -> bug 53433) oo 1353 crypt32:cert -> fixed by MR!1360 (crypt32:cert -> fixed by MR!1360) oo 1356 crypt32:cert -> fixed by MR!1360 (crypt32:cert -> fixed by MR!1360) oo 1357 crypt32:cert -> fixed by MR!1360 (crypt32:cert -> fixed by MR!1360) oo 1359 crypt32:cert -> fixed by MR!1360 (crypt32:cert -> fixed by MR!1360) oo 1358 crypt32:cert -> fixed by MR!1360 (crypt32:cert -> fixed by MR!1360) oo 1347 crypt32:cert -> fixed by MR!1360 (crypt32:cert -> fixed by MR!1360) .. 1360 -> fixes crypt32:cert -> yay fo 1354 crypt32:cert -> fixed by MR!1360 wmvcore:wmvcore -> new bug 54039 (crypt32:cert -> fixed by MR!1360) f. 1315 advapi32:security -> bug 54014 of 1342 crypt32:cert -> fixed by MR!1360 (crypt32:cert -> fixed by MR!1360) (user32:msg -> new bug 54036) (user32:win) oo 1352 crypt32:cert -> fixed by MR!1360 (crypt32:cert -> fixed by MR!1360) .o 1349 (crypt32:cert -> fixed by MR!1360) .. 1348 .. 1260 f. 1346 winediag:d3d_device_create -> lumped with bug 53975 .. 1345 ff 1341 ole32:clipboard -> new bug 54005 (user32:msg -> new bug 54036) (user32:win -> gitlab-specific) f. 1307 d3drm:d3drm -> bug 53975 httpapi:httpapi -> bug 53980 winediag:d3d_device_create -> lumped with bug 53975 .. 1249 .. 1331 f. 1332 d3d8:device -> bug 52983 variant 11-11 -- week 11-07