Patch checking robot coming

[Francois Gouget]

On Sat, 2 Aug 2008, Zachary Goldberg wrote:
[...]
> Its really ironic that you post this today as just yesterday I was
> contemplating the same thing, and not only doing a compile check but
> also a run of the test suite

One problem is that you need an X server, preferably on real hardware to 
please the DirectX chaps. That may be tricky for the chroot, but maybe 
not.

Another problem with running the test suite is that all it's going to 
tell you is that it failed. As I understand it, even on Alexandre's 
machine it fails intermittently.

One way it could work would be to only have the robot complain about new 
failures. This is tricky, see below, but feasible. Let's just take a 
test failure as an example:

   visual.c:506: Test failed: Present failed: Got color 0x00ffffff, expected 0x00ff0000.

The problem here is that if someone adds ten lines near the beginning of 
visual.c, this may turn into:

   visual.c:516: Test failed: Present failed: Got color 0x00ffffff, expected 0x00ff0000.

Or if the colors are random (probably not the case here but happens for 
other tests), then the next time you may get the following message:

   visual.c:506: Test failed: Present failed: Got color 0x001e89df, expected 0x00ff0000.

So how is a bot to know all these are the same error?
With git-blame, that's how. Just use the filename and line number as 
imput to git-blame:

$ git-blame -p -L506,506 dlls/d3d8/tests/visual.c | head -n 1
078523f73e7b708dab06e888c24a1595bb5a63d8 495 506 1

What this means is that this line was first introduced in commit 
078523f7 and was line number 495 at the time. Now, if you commit a 
patch adding 10 lines and then ask about line 516 you will get the exact 
same result. So a unique identifier for this line is:

078523f73e7b708dab06e888c24a1595bb5a63d8:dlls/d3d8/tests/visual.c:495

So all the bot has to do is to run git-blame on each failure line to get 
its unique id, then check the unique id against the list of 
known/allowed (intermittent) failures, and only fail the patch if the 
failure is not in the list.


Now what I'd like is to have something like that for the Windows test 
results...



> and valgrind.

There you have to take into account that for the past week we've had an 
average of 56 patch submissions per day. So if you want the bot to keep 
up it means it must spend less than 25 minutes on each patch. But if you 
want to have a little bit of room for growth it limiting it to 15 
minutes per patch (96 patches/day) would be better...

My understanding is that Valgrinding alone takes much more than 25 
minutes already.

-- 
Francois Gouget <fgouget at free.fr>              http://fgouget.free.fr/
                  Hiroshima '45 - Czernobyl '86 - Windows '95