Direct3D Game test framework

Stefan Dösinger stefan at codeweavers.com
Tue Jun 26 11:19:50 CDT 2007


> We shall discuss the requirements in more exact manner.
>
> 	1. Define demo packages
I think for a start we don't need the ability for users to add new demo 
packages. A fixed set of test packages which is modifyable by raw database 
access would be enough. Otherwise we end up with everyone adding new apps, 
then we are flooded with loads of junk data.

> 	2. Define testing environment configurations
Yep, thats needed for sure.

> The resulting infrastructure shall be human independent in following terms:
> 	human will
> 		prepare a test 'package' (one application)
> 		prepare test units (individual tests)
I'm rather thinking about preparing a set of downloadable test packages. That 
should be modularized though.

> 	machine will
> 		run the test units per package
> 		interpret the results if possible
> 		store the results and run logs (with screenshots)
The machine running the tests and the machine storing the results and 
interpreting them are different in my planning. Also I think we don't want to 
take screenshots when doing performance tests, since taking screenshots is an 
expensive operation. Simmilar for verifying the rendering(which IMHO we 
shouldn't) - that would have to be a secound pass.

I must admit that I never looked very detailed at cxtest, but I thought a bit 
about a possible data model. We would roughly need the following tables:

Users: A username, and whatelse is needed for authentication. We don't want 
anonymous access I think.

Configurations: Testing environemnt(hardware, operating system, etc). Has to 
be creatable/modifyable by the user. One user should be able to use different 
configurations(like 2 different computers running the tests). The configs 
have to be editable without invalidating previous results, but whenever a 
config is edited a test has to be run before and after the modification with 
the same wine version. A possible reason for a config modification is a 
driver update.

Tests: A set of test apps. Can be admin editable only, I'm also happy with 
setting that up with raw database access. A test specifies the application 
and the settings of the app, like 3Dmark2000 800x600, 16 bit color depth. The 
test also has to know what sort of results are returned. Maybe we split this 
table up into base apps(3dmark2000, half life 2, ...) and configs(different 
resolutions, etc).

Concrete Test(or some different name): A configuration + a test + a test 
configuration. It should store some reference result value uppon which we 
decide wether the app should generate a regression warning. Also an optional 
windows comparison value can be stored.

Test result: The result of a test run. An application can provide more than 
one result, like the various tests performed by 3dmark or different hl2 
timedemos. Test results are provieded by the users. Test results have to know 
the wine version used to run the tests, the test and the configuration they 
were run on(ie the concrete test).

Ideally users would download the test packages and run the tests daily, thus 
providing a constant flow of information about our d3d performance, but we 
should also be able to handle manual runs.

Thats just my thought about the requirements. If cxtest has something slightly 
different I'm sure we can adjust the requirements a bit.



More information about the wine-devel mailing list