Francois Gouget : testbot/Engine: Put a VM offline if it repeatedly times out during revert.
Alexandre Julliard
julliard at winehq.org
Fri Sep 20 15:39:35 CDT 2019
Module: tools
Branch: master
Commit: 3f26a883efbff95dde8c25fd6bef67f35bd85e3e
URL: https://source.winehq.org/git/tools.git/?a=commit;h=3f26a883efbff95dde8c25fd6bef67f35bd85e3e
Author: Francois Gouget <fgouget at codeweavers.com>
Date: Fri Sep 20 15:04:53 2019 +0200
testbot/Engine: Put a VM offline if it repeatedly times out during revert.
Putting it in maintenance mode makes it unusable until an administrator
looks at it which could take a long time when quite often the timeout is
just caused by high load on the host.
Now that LibvirtTool 'rate-limits' how often it puts VMs that keep
having errors back online, we can simply put such VMs offline and let
LibvirtTool deal with them.
Signed-off-by: Francois Gouget <fgouget at codeweavers.com>
Signed-off-by: Alexandre Julliard <julliard at winehq.org>
---
testbot/lib/WineTestBot/Engine/Scheduler.pm | 27 ++++++++++++---------------
1 file changed, 12 insertions(+), 15 deletions(-)
diff --git a/testbot/lib/WineTestBot/Engine/Scheduler.pm b/testbot/lib/WineTestBot/Engine/Scheduler.pm
index 569dff7..e9a3ff9 100644
--- a/testbot/lib/WineTestBot/Engine/Scheduler.pm
+++ b/testbot/lib/WineTestBot/Engine/Scheduler.pm
@@ -270,31 +270,28 @@ sub _CheckAndClassifyVMs()
{
# The child process got stuck!
$FoundVMErrors = 1;
- my $NewStatus = "dirty";
if ($VM->Status eq "reverting" or $VM->Status eq "sleeping")
{
my $Errors = ($VM->Errors || 0) + 1;
$VM->Errors($Errors);
- if ($Errors >= $MaxVMErrors)
- {
- $NewStatus = "maintenance";
- NotifyAdministrator("Putting the $VMKey VM in maintenance mode",
- "The last $Errors revert operations timed out.\n\n".
- "No further operation will be attempted until an administrator has put\n".
- "the VM back online.");
- $Sched->{busyvms}->{$VMKey} = 1;
- }
+ $VM->Status("offline");
+ NotifyAdministrator("Putting the $VMKey VM offline",
+ "The last $Errors revert operations timed out.\n\n".
+ "This may be because of some transient load on the VM host but if not\n".
+ "an administrator should look into it. In any case the TestBot will try\n".
+ "to regain access to the VM.");
+ $Sched->{busyvms}->{$VMKey} = 1;
}
- $VM->Status($NewStatus);
- $VM->KillChild();
- $VM->Save();
- $VM->RecordResult($Sched->{records}, "boterror stuck process");
- if ($NewStatus eq "dirty")
+ else
{
+ $VM->Status("dirty");
$Sched->{lambvms}->{$VMKey} = 1;
$Host->{dirty}++;
$Host->{active}++;
}
+ $VM->KillChild();
+ $VM->Save();
+ $VM->RecordResult($Sched->{records}, "boterror stuck process");
}
elsif ($VM->Status =~ /^(?:dirty|running|reverting)$/)
{
More information about the wine-cvs
mailing list