SOC 2009: Application Test Suite

[Jeremy White]

Roderick Colenbrander wrote:
> Hi Austin,
> 
> Not sure if you are aware of it but there is also cxtest which was
> written by codeweavers under the gpl. See http://cxtest.ifne.eu:82/ it
> seems they (still?) use it regulary to track regressions. I haven't
> looked at it and don't know that autohotkey stuff but how do both
> differ? Wouldn't it be better to continue with cxtest? Or perhaps it
> would be better to use autohotkey (assuming it is a widely used app) and
> extend it so that cxtest can be dropped.

Actually, afair, the thinking was that autohotkey would be a nice
complement to cxtest.  That is, if you can write an autohotkey test
that works nicely on a single application on Windows, in theory, you
can drop it into a cxtest frame work and get all of the error tracking
and automated bisecting and fun stuff we've done with cxtest.

Also, I haven't pursued it a lot, but we now have cxtest running
the Mozilla and OpenOffice test suite against Wine.

You can see those results here:
  http://cxtest.ifne.eu:82/wine-error-ratio

They're hard to understand, but the IFNE guys would *love* to
have someone take an interest in it and explain it to them.

The numbers are somewhat encouraging; roughly 50,000 tests run,
and about 97% of them correctly (the majority of the delta
are just unknown).  What's nice is that, in theory, cxtest can
detect any regressions any auto-bisect for guilty patches.

Cheers,

Jeremy