Benchmarking D3D?

[Roderick Colenbrander]

On Sun, Apr 18, 2010 at 5:40 PM, Stefan Dösinger <stefandoesinger at gmx.at> wrote:
>
> Am 18.04.2010 um 04:57 schrieb Dan Kegel:
>> At the moment, my plan is to simply dedicate a machine
>> and script a daily run of a couple benchmarks with +fps,
>> and upload the resulting framerates to a web page
>> so we can notice if somebody checks in something that
>> slows them down.
>> - Dan
> I still have my cxtest scripts and php hackerish that runs a bunch of benchmarks, and I'm still running them. Unfortunately this stuff is a pain to maintain, for various reasons.
>
> I'm testing 3dmark2k, 3dmark2k1, ut2004, halflife2 and team fortress 2. I am also running the Wine tests. The big requirement for the benchmark games is that I have a way to automatically read the results from the games. Running with +fps and collecting the output is imprecise IMO.

There's not much better than can be done. Some games use the same
calculation as in Wine while others take into account previous
results. Here I have adjusted the code to basically: current_fps =
0.1*last_fps + 0.9*frames/(current_time - prev_time) and adjusted the
sample period to 1s. This made the fps a bit more accurate.

> The main problems are:
>
> *) Maintainance. The tf2 timedemo format changes every now and then, and similar issues
> *) The way I wrote the code. Very hacky.
> *) The test machines. I could use a server farm, but I don't have any. The boxes are standing in my living room.
> *) System updates(ubuntu) need to be installed and can affect performance.
> *) Unreliable apps. The steam login fails every now and then, requiring manual intervention.
>

FPS reading requires some hacks to the Wine code anyway e.g. a way to
enable/disable fps dumping. Cxtest or autohotkey can be used to script
the games but yeah it is hard to deal with things like connection
errors.

Roderick