[PATCH] testbot/Engine: Increase VM::errors when a process dies and put it offline.

[Francois Gouget]

A process dying is most likely to be a TestBot bug. But increase the
VM's error count regardless and eventually put it offline. This will
have two benefits:
* If the crash is specific to that VM it will let other VMs runs.
* It will notify the administrator that the VM was put offline and
  that something is wrong.

Signed-off-by: Francois Gouget <fgouget at codeweavers.com>
---
 testbot/lib/WineTestBot/Engine/Scheduler.pm | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/testbot/lib/WineTestBot/Engine/Scheduler.pm b/testbot/lib/WineTestBot/Engine/Scheduler.pm
index 530a2ee1e..b6a89984c 100644
--- a/testbot/lib/WineTestBot/Engine/Scheduler.pm
+++ b/testbot/lib/WineTestBot/Engine/Scheduler.pm
@@ -360,7 +360,18 @@ sub _CheckAndClassifyVMs()
         $FoundVMErrors = 1;
         $VM->ChildDeadline(undef);
         $VM->ChildPid(undef);
-        $VM->Status("dirty");
+        my $Errors = ($VM->Errors || 0) + 1;
+        $VM->Errors($Errors);
+        if ($Errors >= $MaxVMErrors)
+        {
+          $VM->Status("offline");
+          NotifyAdministrator("The TestBot and/or the ". $VM->Name ." VM needs fixing",
+                              "The ". $VM->Name ." VM got $Errors consecutive errors, the last of which is a child process crash which indicates a TestBot bug. But something may also be wrong with the VM for it to have triggered the crash so it has been put offline.\n");
+        }
+        else
+        {
+          $VM->Status("dirty");
+        }
         $VM->Save();
         $VM->RecordResult($Sched->{records}, "boterror process died");
         $Sched->{lambvms}->{$VMKey} = 1;
-- 
2.20.1