Direct3D Game test framework
stefan at codeweavers.com
Mon Jun 25 13:35:48 CDT 2007
> cxtest has already impressive infratructure (installation, creation of
> clean environment) and nice scripting. However, it requires Metacity WM
> and VNC.
The Metacity WM doesn't really concern me, the VNC thing however does. This
makes game testing harder since VNC isn't a part of the average game setup.
We don't have to transfer all the rendering screenshots over VNC though. I am
CCing Martin Pilka on this, maybe he can give us some more insight.
> I'm personally using AutoHotkey(http://www.autohotkey.com) as an
> automatic testing framework backend. It's a Windows application, so it
> can be run through Wine itself and it's sandboxed from X11.
It looks interesting, this would also allow us to run automated tests on
Windows, perhaps to track the effect of driver updates, etc.
> I can imagine Winebot packages to be easily converted to an automatic
> testing engine. Winebot actually covers automated installation and
> creation of clean independent environment.
I don't know if we have to automate the installations as well. Reinstalling
the apps excludes a few factors and makes the tests more repeatable, on the
other hand it is not something that occurs in the average gaming environment.
Not reinstalling can bring up problems related to the game's state on the
disk, like fragmenting game caches. When we see a problem we can reset the
environment manually to exclude the disk content as the cause. Automated game
installation is a nice to have, but not a strict requirement IMHO.
Not reinstalling would ease the use of some controller application that runs
inside wine, since constant installing and removing of the app can leave
garbage in the C drive and the registry over time. On the other hand, an
external application could remove the wineprefix and start with a completely
clean Windows environment.
> Running applications with different log levels, tracing output,
> comparing screenshots with different levels of fuzzy comparaion
> - it's not faraway.
How would fuzzy comparison work?
More information about the wine-devel