Snapshot Cleanup for Failed VMs
When a virtual machine (VM) in a VM backup fails, Portworx Backup deletes the Portworx volume snapshots that were already backed up for that failed VM only, within the same backup cycle. Snapshots of VMs that were backed up successfully in the same backup are not affected. Portworx Enterprise removes the snapshot data from the backup location in the background, so the associated storage is reclaimed after Portworx Backup issues the delete request.
A VM is marked as failed when any of its volumes fails to back up, even if the remaining volumes are backed up successfully. Because a VM is restored as a whole, snapshots of the successfully backed-up volumes cannot be used independently.
In releases earlier than Portworx Backup 3.2.0, snapshots associated with failed or partially successful VM backups were not automatically reclaimed and could accumulate in the backup location. They remained until the corresponding backup was manually deleted. In Portworx Backup 3.2.0 and later, snapshot cleanup begins as soon as the VM is marked as failed and no longer depends on deletion of the backup.
Prerequisites
Snapshot cleanup requires:
-
Portworx Backup 3.2.0 or later
-
Stork 26.5.0 or later on the application clusters
-
Portworx Enterprise 3.7.2 or later on the application clusters
VM backup and restore are not affected if a cluster does not meet these versions. Only the snapshot cleanup behavior changes:
-
Stork earlier than 26.5.0: snapshot cleanup does not run. VM backups continue to work as in earlier releases, and the snapshots of a failed VM remain in the backup location until you delete the backup.
-
Portworx Enterprise earlier than 3.7.2: Portworx Backup can request the deletion, but it cannot reliably confirm that the snapshot data has been physically removed. Cleanup may therefore stay in progress rather than completing. No backup data is lost and no snapshots are orphaned, but the space may not be reclaimed until the cluster is upgraded.
What Portworx Backup cleans up
Snapshot cleanup applies to:
| Item | Support |
|---|---|
| Backup types | VM (KubeVirt) backups only, both manual and scheduled |
| Volumes | Portworx Enterprise backed volumes only |
| Backup locations | All supported backup locations: S3 compatible object store, Azure, Google Cloud, and NFS |
| Backup outcome | Failed and partially successful VM backups |
| Not supported | Namespace backups, volumes that are backed up through CSI snapshots or KDMP, and backups of non-VM applications |
For the items listed as not supported, the existing retention and deletion behavior is unchanged.
Cleanup behavior
-
Cleanup is scoped to the failed VM: only the volumes of a VM that is marked as failed are cleaned up. VMs that were backed up successfully in the same backup, and their snapshots, are left untouched and remain restorable.
-
In-progress snapshot uploads are stopped: If a VM is marked as failed while snapshots of its other volumes are still being uploaded to the backup location, Portworx Backup stops those uploads instead of waiting for them to complete. After an upload reaches the stopped state, Portworx Backup deletes any data already written to the backup location. If an upload is still stopping, deletion is deferred and retried after the upload stops.
-
Volumes shared by more than one VM are retained: if a PVC is attached to more than one VM, its snapshot is retained as long as any VM that uses it was backed up successfully or is still being backed up. The snapshot is removed only after every VM that shares the PVC has failed. This keeps the successful VMs in the backup restorable.
If a shared volume fails to back up, Portworx Backup marks the backup of every VM that uses the volume as Failed. The failed volume displays the CloudSnap error, while the other volumes and affected VM backups display
Aborted: another volume of this VM failed.If a VM backup is already marked as Failed, its status and failure reason remain unchanged, even if another VM sharing the volume fails later.
The overall backup status is Failed if all VM backups fail, or Partial Success if at least one VM backup succeeds and at least one fails.
-
Whole backup failures are cleaned up as well: if all volumes of all VMs were backed up successfully but the backup still fails while the Kubernetes resource definitions are being backed up, the backup cannot be restored. In this case, Portworx Backup deletes the volume data of that backup too.
-
The backup is finalized after the delete request is accepted: Portworx Backup requests Portworx Enterprise to delete the associated cloud snapshots (cloudsnaps) from the backup location. The backup is finalized once Portworx Enterprise acknowledges the request, without waiting for the snapshot data to be physically deleted. If the request cannot be submitted or accepted, for example, when Portworx Enterprise is temporarily unreachable, the backup remains in progress and Portworx Backup retries the request. This ensures that the backup is not finalized until the snapshot deletion request is accepted.
-
Storage space is reclaimed asynchronously: After accepting the deletion request, Portworx Enterprise removes the snapshot data from the backup location in the background. As a result, storage space in the object store or NFS share is reclaimed after the backup is finalized, rather than immediately when the VM is marked as failed or the backup reaches its final state.
How cleanup appears
The following indicators are shown for a failed or partially successful VM backup. They are visible in the Portworx Backup web console and in the equivalent backup details returned by the CLI and the API:
-
The volumes of the failed VM are listed with the status Failed, and their reason includes
Snapshot cleanup issued. This text is appended to the reason that describes why the volume failed, so the original failure reason is preserved. For example:Backup failed for volume: cloudsnap Backup id: <backup-id> for <volume> did not succeed: [Operation stopped by user]. Snapshot cleanup issued -
The size reported for those volumes is zero, because their data no longer exists in the backup location.
-
The total size of the backup accounts only for the data that is retained, that is, the data of the VMs that were backed up successfully.
Use these indicators to confirm that the snapshots of a failed VM were reclaimed.
Deleting a backup location while cleanup is in progress
If you delete a backup location while the cleanup of a failed VM backup is still in progress, Portworx Backup accepts the delete request but does not remove the backup location immediately. The backup location moves to the Delete pending state, and its status reports that it is waiting for volume backup cleanup on the cluster that is running it.
Portworx Backup finalizes the deletion automatically after the cleanup finishes. No action is required from you. For more information on deleting backup locations, see Configure Backup Locations.
Considerations
-
Immutable backup locations: on backup locations with Object Lock or a bucket-level retention policy, the snapshots of a failed VM cannot be removed before the provider-side retention period expires. Portworx Backup still requests the deletion during the backup cycle, but the object store blocks it, so the space is reclaimed only when the retention period expires. Because the cleanup stays outstanding until then, a backup location that you delete in the meantime can remain in the Delete pending state for the remainder of the retention period. For more information, see Retention.
-
Incremental backups after snapshot cleanup: Portworx uses a previous completed snapshot of the same volume as the base for an incremental cloud snapshot. If the base snapshot is reclaimed after a VM fails, the next backup of the affected volume is taken as a full backup. As a result, the next backup might take longer and consume more storage than a typical incremental backup.
-
Restoring a failed VM: a VM that is marked as failed was never restorable, whether or not its snapshots are cleaned up. To back up such a VM again, use Back up Failed VMs. A retry creates a new backup from the source VMs and is not affected by the cleanup of the earlier attempt.
-
Retain count: cleanup reclaims storage space but does not change how backups are counted for retention. Only successful VM backups count toward the retain count of the schedule policy. Failed and partially successful VM backups do not count toward the retain count. Such backups are removed once enough newer successful backups accumulate to move the retention cutoff past them, or when you delete them manually. For more information, see Retention.