Francois Gouget : testbot/Engine: Increase VM::errors when a process dies and put it offline.

[Alexandre Julliard]

Module: tools
Branch: master
Commit: f624b582e35d21eaa01bbcf8e5a72cda3ab3d07a
URL:    https://source.winehq.org/git/tools.git/?a=commit;h=f624b582e35d21eaa01bbcf8e5a72cda3ab3d07a

Author: Francois Gouget <fgouget at codeweavers.com>
Date:   Thu Sep 26 13:35:46 2019 +0200

testbot/Engine: Increase VM::errors when a process dies and put it offline.

A process dying is most likely to be a TestBot bug. But increase the
VM's error count regardless and eventually put it offline. This will
have two benefits:
* If the crash is specific to that VM it will let other VMs runs.
* It will notify the administrator that the VM was put offline and
  that something is wrong.

Signed-off-by: Francois Gouget <fgouget at codeweavers.com>
Signed-off-by: Alexandre Julliard <julliard at winehq.org>

---

 testbot/lib/WineTestBot/Engine/Scheduler.pm | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/testbot/lib/WineTestBot/Engine/Scheduler.pm b/testbot/lib/WineTestBot/Engine/Scheduler.pm
index 530a2ee..b6a8998 100644
--- a/testbot/lib/WineTestBot/Engine/Scheduler.pm
+++ b/testbot/lib/WineTestBot/Engine/Scheduler.pm
@@ -360,7 +360,18 @@ sub _CheckAndClassifyVMs()
         $FoundVMErrors = 1;
         $VM->ChildDeadline(undef);
         $VM->ChildPid(undef);
-        $VM->Status("dirty");
+        my $Errors = ($VM->Errors || 0) + 1;
+        $VM->Errors($Errors);
+        if ($Errors >= $MaxVMErrors)
+        {
+          $VM->Status("offline");
+          NotifyAdministrator("The TestBot and/or the ". $VM->Name ." VM needs fixing",
+                              "The ". $VM->Name ." VM got $Errors consecutive errors, the last of which is a child process crash which indicates a TestBot bug. But something may also be wrong with the VM for it to have triggered the crash so it has been put offline.\n");
+        }
+        else
+        {
+          $VM->Status("dirty");
+        }
         $VM->Save();
         $VM->RecordResult($Sched->{records}, "boterror process died");
         $Sched->{lambvms}->{$VMKey} = 1;