Test framework RFC (Was: Re: CRYPT32/tests: don't crash on win98)

[Jakob Eriksson]

Saulius Krasuckas wrote:

>* On Mon, 14 Nov 2005, Jakob Eriksson wrote:
>  
>
>>* Saulius Krasuckas wrote:
>>    
>>
>>>isNT should be set by testing real behaviour of API, not by using 
>>>GetVersion().  
>>>      
>>>
>>The recommended thing to do is check for something that can hint 
>>of bad things to happen.
>>    
>>
>
>Yes, plus that something and GetVersion() doesn't hint that:
>
>(GetVersion() < 0x80000000) gives 0 both under Win98 SE and WinME.
>Test crashes under Win98SE but it doesn't under WinME.
>I am sure, the versions of crypt32.dll on both boxes are different.
>  
>
Yes, GetVersion() doesn't hint anything.



>I see no API-related tricks, which would help in such cases.  Except for 
>tested DLL version check, maybe.  But that may be too hard, too:
>  
>

Proposal:

Add a mechanism for detecting running a test several times.
The first time serving the purpose of checking if the test crashes or not.
If the test crashes, it is run again, but with an argument telling it that
it crashed last time.

This information is used to hint the test to avoid whatever it feels should
be avoided.

>we should know not only the version of DLL, isNT value and tested function 
>name, but we should also see what parameters and what their values can 
>crash.  This would require a 5-dimensional array of strings, I'd say.
>
>Would anynone here like to maintain such beast?
>  
>

I'm crazy enough, but probably has too little time. So no.

regards,
Jakob