Partial Success in VM Restores
Starting with Portworx Backup 3.2.0, when you restore a backup that contains multiple virtual machines (VMs), Portworx Backup restores and reports the outcome of each VM separately. If some VMs restore successfully while others fail, the restore does not fail as a whole: it completes with a Partial Success status, and the VMs that restored successfully are available for use on the destination cluster.
In releases earlier than Portworx Backup 3.2.0, a VM restore was all-or-nothing: if any volume or resource failed, the entire restore was marked as Failed, none of the VMs were usable, and partially restored resources could be left behind on the destination cluster.
This feature requires Portworx Backup 3.2.0 or later, Portworx Enterprise 3.7.2 or later, and Stork 26.5.0 or later. If any of these components is earlier than the required version, VM restores continue to use the previous all-or-nothing behavior.
Each VM is evaluated independently based on the outcome of its volume and resource restores:
- If all volumes and all associated resources (such as ConfigMaps, Secrets, and NetworkAttachmentDefinitions) of a VM are restored successfully, the VM is marked as Success.
- If any volume of a VM fails to restore, the VM is marked as Failed, and its associated resources are marked as Skipped (they are not restored).
- If all volumes of a VM are restored but one or more of its associated resources fail to restore, the VM is also marked as Failed.
- When a VM is marked as Failed, Portworx Backup cleans up the volumes that were already restored for that VM and marks them as Deleted.
- A failed VM does not affect the other VMs in the same restore. VMs whose volumes and resources all restore successfully are restored completely and are available for use.
Because a VM requires all of its volumes and resources to function correctly, a VM that has one or more failed volume or resource restores is marked as Failed and is not usable. You can continue to use the VMs that restored successfully, and retry only the failed VMs.
To retry, start a new restore and select only the failed VMs. For more information, see Restore specific VMs from a VM backup. When a VM fails during a restore, Portworx Backup removes any successfully restored volumes for that VM and marks them as Deleted. The associated resources are marked as Skipped. If any resources from the previous restore attempt remain on the destination cluster, select Replace existing resources when retrying the restore to overwrite those resources.
Shared resources
Some resources, such as ConfigMaps, Secrets, and volumes, can be referenced by more than one VM. During a partial restore, Portworx Backup handles a shared resource as follows:
- A shared resource is restored if at least one of the VMs that reference it is restored successfully.
- A shared resource is skipped only if all of the VMs that reference it fail.
- If a shared resource itself fails to restore, all of the VMs that reference it are marked as Failed.
Restore success criteria for VM restores
The overall VM restore status is determined by the outcome of the individual VMs included in the restore:
- Success: All VMs included in the restore are restored successfully.
- Partial Success: At least one VM is restored successfully and at least one VM fails.
- Failed: None of the VMs included in the restore are restored successfully.
Using the Retain replace policy does not result in a Partial Success status by itself. If no VM fails, the restore is reported as Success, even when existing resources on the destination cluster are retained instead of being replaced. A restore is reported as Partial Success only when at least one VM succeeds and at least one VM fails.
If you select Replace existing resources during a restore, Portworx Backup deletes the matching existing resources on the destination cluster before it restores them. If a VM then fails to restore, the VM that previously existed on the destination cluster is not recovered. Before you replace existing resources for a critical VM, make sure you have a valid backup that you can restore from.
Per-VM restore status
During a VM restore, Portworx Backup tracks the status of each VM individually. A VM can have one of the following statuses:
- Pending: The VM restore has not started.
- In Progress: The volumes associated with the VM are restored successfully, and the resources associated with the VM are being restored.
- Success: All volumes and associated resources of the VM are restored successfully.
- Failed: At least one volume or resource fails to restore. Associated resources are marked as Skipped, and the volumes that were already restored for the VM are cleaned up and marked as Deleted.
Pending and In Progress are transient states that a VM passes through during the restore; Success and Failed are the final per-VM outcomes.
For a failed VM, Portworx Backup records the reason for the failure on the Restore Details page in the web console, and in the restore CR JSON file. Use this information to identify and address the issue before you retry the restore for the failed VM. For more information, see View restore details.
The per-VM partial success behavior described here applies to VM restores in Classic mode. It does not change how partial success is determined for namespace (application) restores, which continue to report Partial Success based on resource-level outcomes. For more information, see Restore a Namespace Backup.
Example
Consider a restore that includes five VMs:
| Virtual machine | Restore outcome | Per-VM status |
|---|---|---|
| VM-1 | All disks restored and all resources restored | Success |
| VM-2 | All disks restored and all resources restored | Success |
| VM-3 | One disk failed | Failed |
| VM-4 | All disks failed | Failed |
| VM-5 | All disks restored, one resource failed | Failed |
The overall restore is reported as Partial Success because two VMs restore successfully and three VMs fail. VM-1 and VM-2 are usable. Resources associated with VM-3, VM-4, and VM-5 are skipped, and any volumes already restored for these VMs are removed during cleanup.
View restore details
To view the outcome of a VM restore, including the status of individual VMs and the reason for any failures, navigate to the Restore tab. For the required restore, select Vertical ellipsis > Show Details to open the Restore Details page.
If a restore is classified as a large resource restore, the Restore Details page does not display individual resource details. A restore is classified as large when the size of its Restore Custom Resource (restore CR) exceeds the large-resource-size-limit threshold (default 1 MiB), which depends on the size of the CR rather than a fixed number of resources. You can confirm this from the status.largeResourceEnabled field (set to true) in the restore CR JSON. In such cases, review the restore CR JSON file to identify failed resources. For more information, see View Detailed Restore Failures for Large Restores.
Alerts for partial success
Portworx Backup raises a critical alert when a restore fails. A Partial Success restore is a distinct outcome from a failed restore, and restore partial success is not currently listed among the events that generate a critical alert. (For backups, partial success is a listed alert condition.) Because a partially successful restore might not generate an alert, review the Restore Details page after a multi-VM restore to confirm that all VMs were restored. For the full list of events that generate alerts and how to configure email notifications, see Alerts.