Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🐛 Proceed to power off after deprovisioning #1312

Merged
merged 4 commits into from
Sep 15, 2023
Merged

🐛 Proceed to power off after deprovisioning #1312

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7f08b91
Proceed to power off after deprovisioning
honza Jul 12, 2023
5a01d08
Add PoweringOffBeforeDelete state to diagram
honza Aug 25, 2023
abb1d1c
Simplify diagram by removing fake deleting states
honza Sep 14, 2023
7e9dbc5
docs: Add line breaks to long diagram labels
honza Sep 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 7 additions & 9 deletions controllers/metal3.io/host_state_machine.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -536,30 +536,28 @@ func (hsm *hostStateMachine) handleDeprovisioning(info *reconcileInfo) actionRes
hsm.Host.Status.ErrorCount = 0
}
} else {
skipToDelete := func() actionResult {
hsm.NextState = metal3api.StateDeleting
info.postSaveCallbacks = append(info.postSaveCallbacks, deleteWithoutDeprov.Inc)
return actionComplete{}
}

switch r := actResult.(type) {
case actionComplete:
hsm.NextState = metal3api.StateDeleting
hsm.NextState = metal3api.StatePoweringOffBeforeDelete
hsm.Host.Status.ErrorCount = 0
case actionFailed:
// If the provisioner gives up deprovisioning and
// deletion has been requested, continue to delete.
if hsm.Host.Status.ErrorCount > 3 {
info.log.Info("Giving up on host clean up after 3 attempts. The host may still be operational " +
"and cause issues in your clusters. You should clean it up manually now.")
return skipToDelete()
hsm.NextState = metal3api.StatePoweringOffBeforeDelete
info.postSaveCallbacks = append(info.postSaveCallbacks, deleteWithoutDeprov.Inc)
return actionComplete{}
}
case actionError:
if r.NeedsRegistration() && !hsm.haveCreds {
// If the host is not registered as a node in Ironic and we
// lack the credentials to deprovision it, just continue to
// delete.
return skipToDelete()
hsm.NextState = metal3api.StateDeleting
info.postSaveCallbacks = append(info.postSaveCallbacks, deleteWithoutPowerOff.Inc)
return actionComplete{}
}
}
}
Expand Down
7 changes: 6 additions & 1 deletion controllers/metal3.io/host_state_machine_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -997,8 +997,13 @@ func TestErrorCountClearedOnStateTransition(t *testing.T) {
TargetState: metal3api.StateAvailable,
},
{
Scenario: "deprovisioning-to-deleting",
Scenario: "deprovisioning-to-powering-off",
Host: host(metal3api.StateDeprovisioning).setDeletion().withFinalizer().build(),
TargetState: metal3api.StatePoweringOffBeforeDelete,
},
{
Scenario: "powering-off-to-deleting",
Host: host(metal3api.StatePoweringOffBeforeDelete).setDeletion().withFinalizer().build(),
TargetState: metal3api.StateDeleting,
},
}
Expand Down
36 changes: 14 additions & 22 deletions docs/BaremetalHost_ProvisioningState.dot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,57 +5,49 @@ digraph BaremetalHost {

Unmanaged [shape=doublecircle]
Unmanaged -> Registering [label="BMC.* != \"\""]
Unmanaged -> Deleting1 [label="!DeletionTimestamp.IsZero()"]

Deleting1 [shape=point]
Unmanaged -> PoweringOffBeforeDelete [label="!DeletionTimestamp.IsZero()"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This make me worried. Unmanaged hosts cannot be powered off. Isn't it the cause of the issue that we've recently discussed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have a follow up for this here #1356


ExternallyProvisioned [label="Externally\nProvisioned"]

Registering -> Inspecting [label="!externallyProvisioned && NeedsHardwareInspection()"]
Registering -> Preparing [label="!externallyProvisioned && inspectionDisabled()"]
Registering -> Inspecting [label="!externallyProvisioned &&\nNeedsHardwareInspection()"]
Registering -> Preparing [label="!externallyProvisioned &&\ninspectionDisabled()"]
Registering -> ExternallyProvisioned [label="externallyProvisioned"]
Registering -> Deleting2 [label="!DeletionTimestamp.IsZero()"]

Deleting2 [shape=point]
Registering -> PoweringOffBeforeDelete [label="!DeletionTimestamp.IsZero()"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have a follow up for this here #1356


ExternallyProvisioned -> Inspecting [label="!externallyProvisioned && NeedsHardwareInspection()"]
ExternallyProvisioned -> Preparing [label="!externallyProvisioned && !NeedsHardwareInspection()"]
ExternallyProvisioned -> Inspecting [label="!externallyProvisioned &&\nNeedsHardwareInspection()"]
ExternallyProvisioned -> Preparing [label="!externallyProvisioned &&\n!NeedsHardwareInspection()"]
Available -> ExternallyProvisioned [label="externallyProvisioned"]

Inspecting -> Preparing [label="done"]
Inspecting -> Deleting3 [label="!DeletionTimestamp.IsZero()"]

Deleting3 [shape=point]

Deleting5 [shape=point]
Inspecting -> PoweringOffBeforeDelete [label="!DeletionTimestamp.IsZero()"]

Preparing -> Available [label="done"]
Preparing -> Deleting6 [label="!DeletionTimestamp.IsZero()"]

Deleting6 [shape=point]
Preparing -> PoweringOffBeforeDelete [label="!DeletionTimestamp.IsZero()"]

Available [shape=doublecircle]
Available -> Provisioning [label="NeedsProvisioning()"]
Available -> Preparing [label="saveHostProvisioningSettings()"]
Available -> Preparing [label="getHostFirmwareSettings()"]
Available -> Deleting7 [label="!DeletionTimestamp.IsZero()"]
Available -> PoweringOffBeforeDelete [label="!DeletionTimestamp.IsZero()"]
Available -> Inspecting [label="hasInspectAnnotation()"]

Deleting7 [shape=point]

Provisioning -> Provisioned [label=done]
Provisioning -> Deprovisioning [label="failed || !DeletionTimestamp.IsZero()"]
Provisioning -> Deprovisioning [label="failed ||\n!DeletionTimestamp.IsZero()"]

Provisioned [shape=doublecircle]
Provisioned -> Deprovisioning [label="provisioningCancelled()"]
Provisioned -> Deprovisioning [label="!DeletionTimestamp.IsZero()"]

ExternallyProvisioned [shape=doublecircle]
ExternallyProvisioned -> Deleting [label="!DeletionTimestamp.IsZero()"]
ExternallyProvisioned -> PoweringOffBeforeDelete [label="!DeletionTimestamp.IsZero()"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

People have complained about all the Deleting3 [shape=point] stuff making it unclear that the host goes into the Deleting state no matter where it was when deleted.
I wonder if this is the point where it finally becomes completely untenable to hide this info for the sake of saving a few lines on the diagram. How bad does it look if we include all the edges?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something like this:

BaremetalHost_ProvisioningState

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 that's... not outstanding, but maybe not so bad that it's worth the ongoing confusion to avoid it. WDYT?
We don't actually go to PoweringOffBeforeDelete from Unmanaged or Registering, do we?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't actually go to PoweringOffBeforeDelete from Unmanaged or Registering, do we?

We actually do, but the (inevitable) error is handled, so I guess not technically a bug even though it is a bit messy for the user.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I pushed the diagram changes. I think it's reasonably clear.


Deprovisioning -> Provisioning [label="NeedsProvisioning()"]
Deprovisioning -> Available [label="!NeedsProvisioning()"]
Deprovisioning -> Deleting [label="!DeletionTimestamp.IsZero()"]

Deprovisioning -> PoweringOffBeforeDelete
PoweringOffBeforeDelete -> Deleting

Deleting [shape=doublecircle]
}
Binary file modified docs/BaremetalHost_ProvisioningState.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.