Wine Test Bot 1.7.14 analysis

[Jeremy White]

On 03/10/2014 06:08 AM, Henri Verbeet wrote:
> On 10 March 2014 03:13, Jeremy White <jwhite at codeweavers.com> wrote:
>> I wrote a script to analyze winetestbot results on all of the testbot vms
>> except the win8 vms (they are just too broken to try to analyze right now).
>>
>> I'm trying to get a handle on the nature of the bot failures; this current
>> script looks for consistent failures (partly because a consistent failure
>> that goes green is a win, and I badly want to track wins).
>>
>> My results are here:
>>    http://www.winehq.org/~jwhite/ecd24b5a874e.html
>>
> Note that e.g. the win2000 testbot doesn't have results for all runs.
> It looks like this causes the script to classify some failures that
> should be "fixed" as intermittent failures. That might in turn cause
> someone to draw wrong conclusions about e.g. the ddraw tests, if they
> didn't pay enough attention to wine-patches.

Yes; not just win2k, but the win7u bot is unreliable, and one of the 
other win7 bots and one of the vista bots have a few drop outs as well.

But my code, in theory, skips holes in the data, so long as the data 
stays in line.

In other words, a pattern like this:
   SS-FFFF-FF
where S is success, F is failure, and - is missing data, is considered
'fixed'.  A pattern like this:
   F-F--F-F-F
is considered 'consistently failing'.  All other patterns are considered
intermittent.

Note that it's only against the newtb vms; so you'll see the claim that 
d3d9:stateblock is fixed, but there is one non newtb machine where it 
still fails.

Cheers,

Jeremy