TestBot status / todo list

[Francois Gouget]

TL;DR;
- Part 2 of the todo list, focused on the web server side and TestBot 
  reliability.
- False positives caused by intermittent failures should be much rarer.
- Two other false positive sources are close to being fixed.



* Avoid false positives when intermittent failures happen.
  https://bugs.winehq.org/show_bug.cgi?id=47998
  -> Done.

* Provide an efficient way to detect new failures on test.winehq.org
  https://bugs.winehq.org/show_bug.cgi?id=48164
  - This is needed to properly monitor the WineTest results and detect
    regressions so this is a pretty high priority issue.
  -> Now that the intermittent failures are out of the way I plan to
     fix a few small bugs that cause false positives and then work on
     this. (cue jokes about plans and battlefields...)

* Fix handling of child processes.
  - winetest_wait_child_process() is wrong in many ways.
  - But the TestBot also needs changes to detect when a child process
    crashed and avoid generating a false positive.
  -> Wrote WTBS test cases. The Wine and TestBot patches are in
     progress.

* Better handle patch series:
  - Don't mix up different versions of the same patch series
    https://bugs.winehq.org/show_bug.cgi?id=48353
    -> Patches written, need more testing, queued behind other patches.
  - Don't overwrite existing patch series parts
    https://bugs.winehq.org/show_bug.cgi?id=47042
    -> Patch written and tested.

* Automatically update the WinePrefix if a patch requires it
  - The TestBot prepares the WinePrefixes in advance so it does not
    waste time creating them on the fly for each patch. Unfortunately
    some patches require updating the WinePrefix and this causes them
    to fail.
    https://bugs.winehq.org/show_bug.cgi?id=48354
  - The first step is to make use of shared Gecko and Mono installs to
    speed up the WinePrefix creation. Then it may become acceptable to
    create WinePrefixes on the fly.
    https://bugs.winehq.org/show_bug.cgi?id=48652
    -> I have updated wt-daily to support shared Gecko and Mono installs
       (so I know how to do it). Now this just needs to be replicated in
       the TestBot.

* Limit the size of the task reports and logs
  - A recent infinite loop bug in kernel32:process caused it to generate
    140+ MB of test failures. The TestBot then blindly downloaded the
    reports and parsed them. But besides the disk usage on the TestBot
    server this made the impacted job details pretty unusable.
  - So this has shown that it's important for the TestBot to protect
    itself against such issues.
    https://bugs.winehq.org/show_bug.cgi?id=48031
  - This is also a prerequisite to letting developers set $WINEDEBUG on
    their jobs:
    https://bugs.winehq.org/show_bug.cgi?id=47997

* Fix error handling in the WineRun* scripts
  The current situation makes it hard to highlight the testbot.log
  errors on the job details page.
  https://bugs.winehq.org/show_bug.cgi?id=48658

* Add support for com program tests (e.g. chcp.com).
  https://bugs.winehq.org/show_bug.cgi?id=48090
  -> The WTBS has a test case.


-- 
Francois Gouget <fgouget at codeweavers.com>