[ANN] Conformance testing campaign

[Ferenc Wagner]

"Dimitrie O. Paun" <dimi at intelliware.ca> writes:

> On Sat, 27 Sep 2003, Ferenc Wagner wrote:
>
>> For the console test, yes.  I just did not care, because I
>> was promised a better run.
>
> In other words, he is supposed to detect a hanged test,
> and report an error?

Now I lost you here.  Who is he?  The setting is this: if a
test simply crashes, the framework goes on and the failure
will be obvious.  If a test hangs, the user can kill it and
we have the same.  Or the timeout can kill it and maybe note
this in the log.  This all depends on the framework, but we
only have serious trouble if the user kills the framework
itself, which happened in the case of the ME tests.

>> Or simply append a number if the submission is done by
>> email.
>
> For sure we need to append a number, but my point is that
> once we generate that name dinamically, it's ugly and it
> shouldn't be displayed.

An occasional 1, 2 or 3 at the end of your tag?  Why not?

> As for who submitted it, we already have that, so we're
> good.

But it can not possibly be displayed (a name is too wide and
you may have submitted several tests), while it seems useful
for quick visual identication, if you want to concentrate on
a single test run.  Or do you simply not want to do this? I do.

>> But I have a real problem here: which results to put in the
>> main summary if there are many reports for a version?  The
>> one submitted first?  The one with the most/least successes?
>> Or maybe a mixture?  Now it is the one with no tag, which is
>> practically the first submission.
>
> What about the max number of errors in all tests?

Possible, but it would destroy the consistency of the
links in the column.  May be worth it, though.

>>> How come? I thought Jakob includes a dump of the OS version
>>> structure, like so:
>>
>> Yes, he does.  But except for one result submitted by Jakob,
>> all the results are from my .bat-driven zip file.
>
> In other words, for stuff that includes that info, you
> parse it out? Maybe you should stop distributing the
> .bat file, we need to figure out how well do the two
> things work together, no?

Oh yes, I have not got any more reports for a while, and do
not intend to distribute newer versions of my package.  We
had some discussion with Jakob on the format already, but
not finished yet.  Especially not with the master file...

Feri.