Test framework RFC

[Saulius Krasuckas]

* On Mon, 14 Nov 2005, Jakob Eriksson wrote:
> * Saulius Krasuckas wrote:
> > * On Mon, 14 Nov 2005, Jakob Eriksson wrote:
> > > 
> > > The recommended thing to do is check for something that can hint of 
> > > bad things to happen.
> > 
> > (GetVersion() < 0x80000000) gives 0 both under Win98 SE and WinME. 
> > Test crashes under Win98SE but it doesn't under WinME. I am sure, the 
> > versions of crypt32.dll on both boxes are different.
> 
> Proposal:
> 
> Add a mechanism for detecting running a test several times.
> 
> The first time serving the purpose of checking if the test crashes or 
> not. If the test crashes, it is run again, but with an argument telling 
> it that it crashed last time.

Nice shot. :)
But lets look further.

> This information is used to hint the test to avoid whatever it feels 
> should be avoided.

Ok, and this way we check only one crash point.  My patches commented out 
8 such points.  Maybe it's not a problem to rerun every test a dozen of 
times, but where should we put information about every different crash?  
Sounds like a small DB again. :-/

> > we should know not only the version of DLL, isNT value and tested 
> > function name, but we should also see what parameters and what their 
> > values can crash.  This would require a 5-dimensional array of 
> > strings, I'd say.
> > 
> > Would anynone here like to maintain such beast?
> 
> I'm crazy enough, but probably has too little time. So no.

:-]
I wouldn't too.