Direct3D Game test framework

[Stefan Dösinger]

Hi,
Now that we're facing Direct3D 9 feature completion soon - VTF support coming 
in from Henri, High order patches and thread safety from me - the d3d work 
will change a bit soon. Instead of adding new features we'll debug games and 
improve performance. It would be good if we had some automated testing 
framework to support the process :-)

Unfortunately this isn't easy. Games do not render exactly, so verifying that 
a game doesn't show any visual regressions automatically may be more effort 
than reward. Here I think we'll have to rely on users testing the games 
regularly and testing regressions.

I am more concerned about the performance thing. It is very easy to make some 
performance improvement that does more harm than good. Also benchmarks take 
quite some time and are affected by many external parameters. So I'd like 
some automated and controlled performance testing environment. Instead of 
building one from scratch it would be cool if we could reuse existing 
software.

cxtest comes to my mind(www.cxtest.org). It is a test suit built by 
codeweavers for running regression tests on Windows apps. It can remote 
control applications using keyboard input and compares screenshots to 
reference ones. It could be used to start benchmarks automatically. A plus 
here is that some machines run nightly tests. However, those are VMs, so no 
3D acceleration. A problem is also reading back the results. Does anyone know 
some applications for doing that? Windows or Linux apps, commercial or open 
source ones(prefered, though).

Then we need benchmarking apps. Here regular Windows benchmarks come to my 
mind(3DMarkXYZW), games with benchmark facilities(hl2, ut, ...). We can 
record tests, get a reference performance counter from windows and run 
nightly tests(or manually started tests). A problem we're facing is reading 
back the results. Taking screenshots and running OCR on them is error prone, 
we need a way to get the numbers from the benchmarks, not an image of the 
numbers.

Now we need a concrete collection of benchmarks. A mixture of popular 
Benchmarks like 3DMark would be good, and some more cientific benchmarks. So 
far I have:

3DMark:
Popular benchmark for testing the overall performance, doesn't give very 
detailed results though.
Controlling: Standard windows controls, keyboard events should do the job
Result readback: Can write results to a file, but must take care about the 
licensing. Not everything is free, most likely a license is needed.

Half-Life 2:
Nice facility of recordable timedemos. Can record average game demos, but also 
demos which stress specific features.
Controlling: Command line parameters
Result readback: Can be asked to copy the console into the clipboard

UT2004:
Popular engine. Has a Linux port, but many mods only ship the d3d renderer.
Controlling: Unknown
Result readback: unknown.

DirectX sdk demos, nvidia / ati demos:
Stress specific rendering features. Good for testing performance(with vsync 
removed), or a bad idea?
Controlling, Readback: Source code is available and can be modified. License, 
especially possible benchmark clauses?

Other suggestions?

Another nice-to-have thing would be a game independent profiling method, 
simmilar to the +fps debug channel, just more detailed. Some games do not 
have built-in benchmarking, it would still be nice to use them. That means 
facing some new problems? How to tell loading from in-game rendering? How to 
remove control those games in a repeatable way, especially if there are some 
random elements in the game? More detailed data from wined3d could also be 
used to draw nice graphs for other benchmarks :-)

We'll need some hardware diversity too. It would be nice if the tests worked 
on Linux and MacOS, and a variety of hardware was used. I want at least 
nvidia, ati and intel gpu's, older and newer ones. I could also revive my ATI 
mach64 :-)

Any other comments or suggestions? I think once I'm done with the current exam 
season and my patch backlog is in I will retire from direct wined3d 
development for a few days / weeks to set up a testing environment, then 
start fixing games.