Direct3D Game test framework

[Michal Okresa]

Stefan Dösinger wrote:
>> We shall discuss the requirements in more exact manner.
>>
>> 	1. Define demo packages
>>     
> I think for a start we don't need the ability for users to add new demo 
> packages. A fixed set of test packages which is modifyable by raw database 
> access would be enough. Otherwise we end up with everyone adding new apps, 
> then we are flooded with loads of junk data.
>
>   
>> 	2. Define testing environment configurations
>>     
> Yep, thats needed for sure.
>
>   
>> The resulting infrastructure shall be human independent in following terms:
>> 	human will
>> 		prepare a test 'package' (one application)
>> 		prepare test units (individual tests)
>>     
> I'm rather thinking about preparing a set of downloadable test packages. That 
> should be modularized though.
>
>   
>> 	machine will
>> 		run the test units per package
>> 		interpret the results if possible
>> 		store the results and run logs (with screenshots)
>>     
> The machine running the tests and the machine storing the results and 
> interpreting them are different in my planning. Also I think we don't want to 
> take screenshots when doing performance tests, since taking screenshots is an 
> expensive operation. Simmilar for verifying the rendering(which IMHO we 
> shouldn't) - that would have to be a secound pass.
>
> I must admit that I never looked very detailed at cxtest, but I thought a bit 
> about a possible data model. We would roughly need the following tables:
>
> Users: A username, and whatelse is needed for authentication. We don't want 
> anonymous access I think.
>
> Configurations: Testing environemnt(hardware, operating system, etc). Has to 
> be creatable/modifyable by the user. One user should be able to use different 
> configurations(like 2 different computers running the tests). The configs 
> have to be editable without invalidating previous results, but whenever a 
> config is edited a test has to be run before and after the modification with 
> the same wine version. A possible reason for a config modification is a 
> driver update.
>
> Tests: A set of test apps. Can be admin editable only, I'm also happy with 
> setting that up with raw database access. A test specifies the application 
> and the settings of the app, like 3Dmark2000 800x600, 16 bit color depth. The 
> test also has to know what sort of results are returned. Maybe we split this 
> table up into base apps(3dmark2000, half life 2, ...) and configs(different 
> resolutions, etc).
>
> Concrete Test(or some different name): A configuration + a test + a test 
> configuration. It should store some reference result value uppon which we 
> decide wether the app should generate a regression warning. Also an optional 
> windows comparison value can be stored.
>
> Test result: The result of a test run. An application can provide more than 
> one result, like the various tests performed by 3dmark or different hl2 
> timedemos. Test results are provieded by the users. Test results have to know 
> the wine version used to run the tests, the test and the configuration they 
> were run on(ie the concrete test).
>
> Ideally users would download the test packages and run the tests daily, thus 
> providing a constant flow of information about our d3d performance, but we 
> should also be able to handle manual runs.
>
> Thats just my thought about the requirements. If cxtest has something slightly 
> different I'm sure we can adjust the requirements a bit.
>   
Hi Stefan,

CxTest has simpler data model than one described above. For a long time 
CxTest is general framework for testing applications not games.

We (people around CxTest) think that extending CxTest is better approach 
to achieve your goals rather than writing new application or modifying 
existing. It could be done with smaller effort (in opposite of 
AutoHotkey, AutoIt, ...) because CxTest is  specialised for testing Wine 
and CrossOver and it can deal with wine's specifics. One of big 
advantages is that result's web page exists and it is collecting reports 
from nightly tests.

For the start we can set up new evaluation page on the CxTest's page 
http://www.cxtest.org/evaluation. System under test (SUT) will send 
report  which will be interpreted by server as game performance results. 
For achieving this SUT will also send XML file with processing 
instructions. For example: attachment result1.raw contains data for 2D 
graphs and so on.

CxTest core needs some modifications because it is extensively using 
screen shots to detect responsibility of tested application. 
Implementation of new functions for test scripts is also possible.

Note, if you are experimenting with cxtest, I strongly recommend to you 
to use cvs version of CxTest because cxtest.sh script is not often 
updated. And I have checked 3DMark package from you. It is OK, but for 
better reliability of tests use --window "window name" parameter for 
send_keystroke. In cvs version fvwm is used instead of metacity and 
--window parameter is essential to achieve  best results. For ID's of 
windows, buttons, ... you may use wpickclick/pickclick from utils directory.

--
Michal Okresa