do ERR messages imply bugs?

[chris ahrendt]

Ben Klein wrote:
> 2009/4/13 chris ahrendt <celticht32 at aol.com>:
>   
>> Is there a guide documenting what each test is supposed to do etc?
>>     
>
> Source code.
>
> Before you say that's an unacceptable answer, the sheer number of test
> cases (especially considering those that keep getting added) would
> make it impractical to the point of impossibility to maintain proper
> documentation on each test. And when the tests are only intended for
> developers anyway, the source code is perfectly suitable as
> documentation. The tests are relatively simple, and it's clear which
> case is being test by the use of ok().
>
>   
>> I run the test set to see where the current version of wine is on any of my
>> particular hardware.
>>     
>
> Then you should learn to interpret the ERRs correctly.
>
>   
>> I am not allowed to develop wine due to NDA's , and so forth I am under. And
>> have discussed
>> with the Alexandre and Dan concerning these NDA's and what I can do to help.
>>
>>     
>>>> That does not make any sense...  There has to be a consistent way for
>>>> developers and users
>>>> to report or work on issues..
>>>>
>>>>         
>>> It's called bugzilla.
>>>
>>>       
>> I believe you may be missing my point Ben. By consistent I mean an ERR means
>> this and only means this. Warning, info etc...
>>     
>
> As far as I'm concerned, that already exists, even if it's not written
> explicitly and finitely in the dev guide. ERRs caused by the test
> suite can be ignored as long as the tests run correctly. By the same
> logic, ERRs caused by real applications can be ignored as long as the
> apps still run correctly. In the former case, there is no motivation
> for the devs to fix them. In the latter, there is minimal motivation,
> and the devs have bigger problems to worry about than things that
> aren't actually *breaking* anything at the moment.
>
>   
>> From what I  have been following here and also seeing in the code an error
>> doesn't always follow the
>> coding standard set forth in the developers guide.
>>     
>
> I don't think you understand the nature of the test suite. There are
> some tests that are 100% valid logic and are expected to create no
> ERRs or WARNs. These are called positive cases. There are some tests
> that are not 100% valid logic, and do things wrong deliberately, and
> are expected to create both ERRs and WARNs. That's an "and" there, not
> an "or". Both ERRs and WARNs. This cannot change.
>
>   
>>>> and like Vincent says the use has gotten a little skewed.
>>>>         
>>> Or been misinterpreted ... though a review of the developers' guide
>>> description of ERR wouldn't hurt.
>>>
>>>       
>> I would agree it may be... and maybe a review is in order..
>>     
>
> So who wants to volunteer? :)
>
>
>   
Not I =)


> 2009/4/13 chris ahrendt <celticht32 at aol.com>:
>   
>> An Excellent point was mad in one of the bugs I reported :
>>
>> Comment #25 from Austin English <austinenglish at gmail.com>
>>
>> The problem is that this isn't a 'normal' application doing weird things.
>> It's our testsuite, which does some _really_ strange stuff, e.g., lots of
>> corner case testing.
>> Our implementation code, however, knows this is weird, and prints an error.
>> The problem is that there's no way to tell, e.g, dlls/mshtml/htmldoc.c that
>> what's
>> being tested is dlls/mshtml/tests/doc.c, rather than foobar.exe. It's a bit
>> misleading, which makes it a bit harder for users.
>>     
>
> This is EXACTLY what I've been saying. You ignore me, and listen to
> Austin. Absolutely fantastic.
>
>   
not AT All ben =D  I do and have understood that there are positive and 
negative test cases.
The point I guess I am trying to make is in the test cases where there 
is an expected failure just
print something out saying this is expected. Not Complicated at all.
>> So, there has to be a happy middle in this...
>> why not have in the test suite some printf or the like that says to the
>> effect that err output is to be expected or the like.. Most users if they
>> see ERROR on the screen
>> be it in a test case or an application will think its a bug and not just
>> ignore it.
>>     
>
> Most users DON'T RUN THE TEST SUITE. It's not useful for users, it's
> ONLY useful for developers, and developers know how to interpret these
> ERRs correctly. You don't.
>   

Thanks for the attack ben... again you miss the point.. alot more users 
do run the tests than you may think.
People evaluating the code for use in their enterprise will run the test 
sets.. and if they don't complete or run
clean than they don't implement.

C