RFC: Allow marking unreliable tests as flaky.

[Francois Gouget]

On Wed, 1 Jun 2022, Esme Povirk (she/they) wrote:

> It might be better to use something like broken() to encourage more
> precision in marking the failures. For example, if something fails
> with ACCESS_DENIED sometimes, we'd still like to know if it starts
> failing with FILE_NOT_FOUND.

So something like this?

  ok(success || flaky(gle == ACCESS_DENIED), "Call failed: gle=%lu\n", gle);

I suspect we'd get a lot of "|| flaky(1)", for instance for all the 
"always new" error messages.

Also the goal is to make the tests more "reliable" but if we are too 
specific about which unexpected conditions we allow there's a risk we'll 
spend our time expanding the condition inside the flaky() call. But 
maybe it would be okay to give it a few tries and fall back to flaky(1).


-- 
Francois Gouget <fgouget at codeweavers.com>