Hey all,

I thought I would start my annual harangue a little bit early this year. I thought I'd summarize the lens I see things with and then see if there is anything constructive we can do now, and then again when we meet in person. Then we'll also have drink to turn to if we end with nothing but despair.

The ideal is that this page: http://test.winehq.org/data/ be covered entirely in green. That would indicate that our unit tests ran successfully on all tested systems. A further ideal is that it would have 'Mac' and 'Android' columns that were also green. The holy dream we all crave is that a 'make test' would work in a rational fashion. And the ultimate fantasy is that every patch sent would be tested not only on Windows if the tests change, but on Linux, Mac, and Android as well. (Alexandre does that by hand now, but it'd be nice to automate that test).

A more reasonable ideal is that all of our tests would run successfully on a well curated list of 'rigorous' test machines. I don't believe that we have an official page for that; I maintain an unofficial one here: https://www.winehq.org/~jwhite/latest.html That is all of the 'newtb' Windows VMs excluding Windows 2000, Windows 8, and Windows 10.

That list of failures has fluctuated; starting at about 40, and once getting down to as few as a dozen. It now stands at about 20, where it's been for a while. Nicely (?), we're down to only intermittent failures :-/.

I think the instinct has been to fix all of the Windows tests; that once those are consistently green, then it would make sense to go after a well defined Linux rig and push it to green.

CodeWeavers has a rack of hardware and we're happy to put any flavor of system in there (and have done so, with standard rigs for testing AMD and NVidia Linux boxes). A quick scan suggests that those rigs stand at about the same number of failures; low to mid teens.

I think we are lacking several things:

1. Some help for Francois. He's basically doing all of this on his own. We could use some people willing to fight through Perl to help extend our capabilities.

2. The ability (will?) to drive the Windows tests to green. Is it time to articulate a kind of test that is expected to periodically fail? In other words, do we have tests that 'reasonably' fail, and so we should redefine that failure?

Many years in the past, we had constructive sessions where I forcibly prevented people from going to the pub until we flipped a whole lot more bits to green. I can remember times when that worked quite nicely. But we haven't been as productive with that lately. Is it worth dedicating another block of time to it at the Wine conference? Would it work better if we prepared ahead of time?

For example, let me suggest this: that developers start right now running winetest on machines they plan to bring to Wineconf. Then we can have that body of information available to us at the conference, and we can choose to attack issues that show commonality.

Does that make sense?

Any other prep work we should do? (Jacek and Piotr assure me that there will be plenty of beer for drowning sorrows, so that seems to be covered).

Cheers,

Jeremy