RFC How to get rid of "always new" TestBot false positives?

[Francois Gouget]

On Fri, 27 Mar 2020, Zebediah Figura wrote:

> Okay, so, some thoughts:
> 
> (1) How many of these are there? Have you compiled a list? Maybe we should
> just bite the bullet and add broken() for them instead, and defer figuring out
> why it's actually broken for later.

I'll start with that part.

I only checked the failures that happen in the base 32 bit test 
configurations so far (anything that's highlighted in "32 bit WineTest: 
base VMs" jobs). A brief look leads me to expect finding essentially the 
same set of issues in the 64 bit tests ("642 bit WineTest: base VMs" 
jobs), but the Wine jobs may have quite a different set ("WineTest: 
debiant VM" jobs)

When attacking the "always new" failures I think the top priority is 
with those:

* comctl32:datetime: Fails in the Arabic locale
  datetime.c:796: Test failed: Expected 24/03/2020, got 2020/03/24
  https://bugs.winehq.org/show_bug.cgi?id=47866

* user32:cursorinfo "wrong info cursor" failure in Chinese locale on Windows 10
  cursoricon.c:2366: Test failed: wrong info cursor 003C04BB/00010003
  https://bugs.winehq.org/show_bug.cgi?id=48808

* user32:win "unexpected 0x738 message" Windows 10 failures
  win.c:3858: Test failed: message 0738 available
  win.c:3852: Test failed: hwnd 00210500 message 0738
  https://bugs.winehq.org/show_bug.cgi?id=48815

* user32:win "unexpected 0x0282 message" Chinese Windows 10 failures
  win.c:3849: Test failed: hwnd 00660590 message 0282
  win.c:3928: Test failed: hwnd 00660590/01D90478 message 0282
  win.c:3931: Test failed: hwnd 00660590/01D90478 message 0282
  https://bugs.winehq.org/show_bug.cgi?id=48819

* user32:win "unexpected 0x7fff message" Japanese Windows 10 failures
  win.c:3849: Test failed: hwnd 00660590 message 7fff
  win.c:3928: Test failed: hwnd 00660590/01D90478 message 7fff
  win.c:3931: Test failed: hwnd 00660590/01D90478 message 7fff
  https://bugs.winehq.org/show_bug.cgi?id=48820

* Error dialogs cause the user32:winstation tests to fail
  winstation.c:973: Test failed: unexpected foreground window 00070064
  https://bugs.winehq.org/show_bug.cgi?id=47894

* dxgi:dxgi
  dxgi.c:5185: Test failed: Got unexpected message 0x31f, hwnd 003101F4, wparam 0x1, lparam 0.
  dxgi.c:5185: Test failed: Got unexpected message 0x46, hwnd 003101F4, wparam 0, lparam 0x28fc30.
  dxgi.c:5185: Test failed: Got unexpected message 0x24, hwnd 003101F4, wparam 0, lparam 0x28f998.

* kernel32:debugger
  debugger.c:142: Test failed: unable to open 'C:\Users\winetest\AppData\Local\Temp\wt2983.tmp'
  debugger.c:142: Test failed: failed to open: C:\Users\winetest\AppData\Local\Temp\wt2983.tmp
  debugger.c:654: Test failed: the child and debugged pids don't match: 4780 != 6452
  -> These are actually quite rare (12 instances in all of 
     test.winehq.org's reports).


I'm deferring investigating the following two because it looks like they 
may be caused by the "big delay" issues. So I think it makes more sense 
to revisit them after I've had a look at it from the QEmu side.

* kernel32:comm
  comm.c:886: Test failed: WaitCommEvent used 1594 ms for waiting
  The cause for this one may have more to do with the lack of 
  serial baud rate emulation in QEmu.
  https://bugs.winehq.org/show_bug.cgi?id=48108

* mmdevapi:render
  render.c:1025: Test failed: Position 6592 too far after playing 100ms


The remaining new failures in that TestBot report look to be more in the 
"rare failure" category. Sometimes the issue is made worse by the lack 
of reference results due to WineTest timing out (e.g. w1064v1809-he, the 
TestBot cannot currently use incomplete test reports but I got an idea 
for solving that).



-- 
Francois Gouget <fgouget at codeweavers.com>