View Detailed Restore Failures for Large Restores
For large resource restores, detailed resource information is not retained in the restore object or displayed in the Portworx Backup web console. For a partial restore, Portworx Backup stores a copy of the Restore Custom Resource (restore CR) as a JSON file in the backup location. You can review this file to identify the status of individual resources and the reasons for any failures.
Starting with Portworx Backup 3.2.0, you do not need direct access to the backup location to retrieve the restore CR file. From the Portworx Backup web console or CLI, you can retrieve a temporary download URL for object store backup locations, or the NFS path and mount command for NFS backup locations.
For more information, see Retrieve the restore CR file from the web console and Retrieve the restore CR file using the CLI.
- A partial restore status indicates that one or more resources failed during the restore process. It does not indicate that resources were skipped or retained due to user-defined retain policies.
- To verify whether a non-large backup is treated as a large resource restore, check the Stork logs for the following message:
You can also verify this by checking the corresponding flag in the restore object.Stripping all the resource info from restore cr as it is a large resource based restore
When the restore CR file is available
Portworx Backup stores a restore CR file in the backup location only when all of the following conditions are met:
- The restore is treated as a large resource restore. This happens when the size of the restore CR exceeds the
large-resource-size-limitthreshold, which defaults to 1 MiB and is set in thestork-controller-configConfigMap on the application cluster. A restore of a backup that is not itself large can still be treated as a large resource restore. For more information, see stork-controller-config ConfigMap parameters. - The restore reaches a terminal state of Partial success or Failed.
- At least one resource fails to restore. A restore that reports Partial success only because resources were skipped or retained does not produce a restore CR file. For example, this can occur when a resource already exists in the destination namespace and the replace policy is set to retain it.
- The restore is performed in Classic mode. Restores performed in Federated mode do not produce a restore CR file, and the retrieval options described in this topic are not available for these restores.
If any of these conditions is not met, there is no restore CR file to retrieve. In this case, the web console and CLI indicate that a download location is not available for the restore.
Restore CR file storage location
Depending on the backup location type, the restore CR file is stored in:
-
Object store (S3 and S3-compatible, Azure Blob, and Google Cloud Storage):
<bucket-name>/restore-crs/<restore-cr-name>-<shortUID>.json -
NFS:
<export-path>/<sub-path>/restore-crs/<restore-cr-name>-<shortUID>.jsonIn NFS locations,
<sub-path>refers to the optionally configured subdirectory within the export path.importantThe restore CR file is deleted after you delete the corresponding restore object using the Portworx Backup UI or CLI. For object lock–enabled backup locations, the restore CR file is deleted automatically after the retention period expires.
Retrieve the restore CR file from the web console
For any restore that has a restore CR file stored in the backup location, as listed in When the restore CR file is available, you can retrieve that file directly from the Portworx Backup web console, without requiring direct credentials for the backup location.
Requirements
Ensure that the following requirements are met:
- Portworx Backup 3.2.0 or later is installed. If the backup location is configured with an encryption key, Stork 26.5.0 or later must have been installed on the application cluster when the restore was performed. For more information, see Considerations and limitations.
- You must have access to both the restore and its associated backup location in Portworx Backup, either as the owner or through sharing. No additional roles or permissions are required to retrieve the restore CR file. If you can view the restore but do not have access to its backup location, the Download Resources option remains visible, but retrieval fails with an error indicating that the backup location is missing.
- The backup location is registered in Portworx Backup. If the backup location was deleted after the restore completed, register it again with the same name before retrieving the file.
- The system from which you retrieve the file can reach the backup location endpoint, such as the object store bucket endpoint or NFS server. Portworx Backup does not proxy the file contents. The file is downloaded or copied directly from the backup location to your system.
Procedure
To retrieve the restore CR file from the web console, follow these steps:
-
Log in to Portworx Backup web console.
-
From the left navigation pane, click the Clusters icon
. -
On the Clusters page, click the application cluster on which you performed the restore.
-
Click the Restore tab, and expand the row of the required restore.
For a large resource restore that ended with Partial success or Failed, Portworx Backup displays an information banner that indicates resource details are not available for large restores. -
In the banner, click Download Resources.
The system displays the Custom resources path window with the name of the selected restore, the backend type of the backup location (Object Store or NFS), and the location details of the restore CR file. -
Depending on the backend type, complete one of the following:
-
Object Store: copy the download URL using the copy icon, and then use it in either of the following ways from a system that can reach the bucket endpoint:
-
Paste the URL into the address bar of a web browser to view the contents of the restore CR JSON file directly in the browser, without saving it first.
-
Pass the URL to a tool such as
curlorwgetto save the file locally:curl -o restore-cr.json "<copied-url>"
The window also displays the object path of the file within the bucket, which you can copy separately.
-
-
NFS: Copy the displayed command using the copy icon, and run it on a Linux system that can access the NFS server. The command creates a unique temporary mount point, mounts the NFS export, copies the restore CR file to
/tmp/restore-cr.json, unmounts the export, and removes the temporary directory.MOUNT_DIR=$(mktemp -d) && \sudo mount -t nfs -o <mount-options> "<nfs-server>:<export-path>" "$MOUNT_DIR" && \cp "$MOUNT_DIR/<file-path>" /tmp/restore-cr.jsonRESULT=$?if mountpoint -q "$MOUNT_DIR"; thensudo umount "$MOUNT_DIR" || exit 1firmdir "$MOUNT_DIR"if [ "$RESULT" -ne 0 ]; thenexit "$RESULT"finoteThe command uses
mktemp -dto create a unique temporary mount point, preventing the NFS export from being mounted over an existing file system. It also unmounts the export and removes the temporary directory after the copy operation.For other operating systems, use the displayed server, export path, and file path values with your own tooling to mount the export and copy the file.
-
-
Open the retrieved file and review the per-resource status, as described in Identifying failed resources in large restores.
Retrieve the restore CR file using the CLI
You can use the CLI to retrieve the restore CR file for an eligible restore. For the conditions that determine whether a restore CR file is available, see When the restore CR file is available.
For the command syntax, available options, supported values, and usage examples, see pxb get restore-cr in the Portworx CLI documentation.
Considerations and limitations
Observe the following before you retrieve the restore CR file:
- Network reachability: your system must be able to reach the backup location endpoint. Portworx Backup issues the download URL or the NFS access details, but the file itself is transferred directly between your system and the backup location. If your system cannot reach the endpoint, use a host that can, such as a bastion host, or connect through a VPN or an SSH tunnel.
- URL lifetime: the download URL for object store backup locations is temporary. It is valid for 30 minutes (
1800seconds) by default, and you can request a different lifetime with the--expiry-secondsoption of the CLI, within a supported range of60seconds (1 minute) to604800seconds (7 days).
If you request a value outside this range, Portworx Backup adjusts it to the nearest supported value and does not notify you of the change. For example, a request for30seconds results in a URL that is valid for 60 seconds, and a request for 30 days results in a URL that is valid for 7 days. Check the expiry time that the web console or CLI reports for the URL to confirm the lifetime that was applied.
If the URL expires before you use it, retrieve a new one. NFS access details do not expire, because access to an NFS export is controlled by the NFS server. - Treat the URL as a credential: anyone who has the download URL can download the restore CR file until the URL expires, without signing in to Portworx Backup. Do not paste it into support tickets, chat messages, or shell history, and do not store it in logs. The URL cannot be revoked before it expires. If you must revoke access immediately, rotate the backup location credentials at the storage provider.
- Encryption: starting with Portworx Backup 3.2.0 and Stork 26.5.0, the restore CR file is stored unencrypted in the backup location, even when the backup location is configured with an encryption key. All other backup data, such as volume snapshots and resource archives, continues to be encrypted according to the backup location configuration. The restore CR file contains only Kubernetes resource metadata and per-resource restore status; it does not contain secret data or storage credentials.
Restore CR files that were written to an encrypted backup location by an earlier Stork version remain encrypted and cannot be retrieved with the options described in this topic. Retrieve those files directly from the backup location and decrypt them manually with the backup location encryption key. - File size: Portworx Backup does not serve restore CR files larger than 1 GiB. You can change this limit with the
MAX_RESTORE_CR_DOWNLOAD_SIZE_BYTESparameter of thepx-backup-configConfigMap. For more information, see px-backup-config ConfigMap parameters. - Federated mode: the Download Resources option is not available for restores performed in Federated mode, because Portworx Backup does not store a restore CR file for those restores.
Troubleshoot restore CR retrieval
The following table lists the common messages that you may encounter while retrieving a restore CR file, and the corresponding resolution:
| Message | Cause and resolution |
|---|---|
| A download location is not available for this restore. | The restore does not have a restore CR file in the backup location. Verify the conditions listed in When the restore CR file is available. |
| The restore is not in a terminal state. | The restore is still in progress. Wait for the restore to complete, and then retry. |
| The restore succeeded, so per-resource restore CR files are not uploaded. | Portworx Backup stores a restore CR file only for restores that end with partial success or failure. A fully successful restore has no restore CR file. |
| No restore CR file was uploaded for this restore. | The restore did not meet the conditions that cause Portworx Backup to store a restore CR file, or the file was removed from the backup location, for example by a bucket lifecycle policy. Exclude the restore-crs/ prefix from any lifecycle policy that you apply to the bucket. |
| The backup location is missing. | The backup location associated with the restore was deleted after the restore completed. If the underlying bucket or export still holds the file, register the backup location again with the same name and retry. |
| The restore CR file appears to be encrypted. | The file was written to an encrypted backup location by an earlier Stork version. Retrieve the file directly from the backup location and decrypt it manually with the backup location encryption key. |
| The restore CR file exceeds the maximum download size. | The resource inventory of the restore is unusually large. Increase the MAX_RESTORE_CR_DOWNLOAD_SIZE_BYTES value in the px-backup-config ConfigMap, or contact Portworx support. |
| HTTP 403 or a signature error when you use the download URL. | The URL expired before you used it. Retrieve a new URL, and if required, request a longer lifetime with the --expiry-seconds option. |
| A TLS or host name error when you use the download URL. | The object store endpoint uses a certificate that your system does not trust. Use the --bucket-ca-cert option of the CLI to trust the certificate authority of the endpoint, or add the certificate authority to the trust store of your system. |
mount.nfs: access denied or the mount command hangs. | The NFS server does not allow mounts from your system, or the server is not reachable. Verify that the IP address of your system is allowed in the export configuration of the NFS server. |
Identifying failed resources in large restores
To identify the failed resources, follow these steps:
-
Open the restore CR JSON file and locate the following section:
status.resources[]Each resource entry includes a
statusandreasonfield:-
If status is Successful, the resource was restored successfully.
-
If status is Failed, review the reason field to identify the exact error.
-
-
Check for the following
reasonstring for the"status": "Failed"."Error applying resource: STUB: simulated failure for resource test-secret9 as configured in stubResourcesToFail"The Error applying resource indicates the resource failed to restore in a large resource restore.
Sample restore CR JSON
This is a sample JSON file for documentation purposes only to show the successful and failed resources.
{
"metadata": {
"name": "n2-85d1556",
"namespace": "kube-system",
"uid": "898XXXXX-abcd-cbcd-8b16-f03a59c0b77f",
"resourceVersion": "329877",
"generation": 6,
"creationTimestamp": "2026-01-28T03:58:59Z",
"labels": {
"app.kubernetes.io/part-of": "px-backup"
},
"annotations": {
"portworx.io/cluster-name": "local-clu",
"portworx.io/cluster-uid": "fceXXXXX-abcd-abcd-b3fb-ed972fdXXXXXX",
"portworx.io/created-by": "px-backup",
"portworx.io/last-update": "2026-01-28T03:58:59.055573928Z",
"portworx.io/org-id": "default",
"portworx.io/restore-name": "n2",
"portworx.io/restore-uid": "85d1XXXXX-abcd-abcd-86ff-72f3e397d430"
},
"finalizers": [
"stork.libopenstorage.org/finalizer-cleanup"
],
"managedFields": [
{
"manager": "px-backup",
"operation": "Update",
"apiVersion": "stork.libopenstorage.org/v1alpha1",
"time": "2026-01-28T03:58:59Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:portworx.io/cluster-name": {},
"f:portworx.io/cluster-uid": {},
"f:portworx.io/created-by": {},
"f:portworx.io/last-update": {},
"f:portworx.io/org-id": {},
"f:portworx.io/restore-name": {},
"f:portworx.io/restore-uid": {}
},
"f:labels": {
".": {},
"f:app.kubernetes.io/part-of": {}
}
},
"f:spec": {
".": {},
"f:backupLocation": {},
"f:backupName": {},
"f:backupObjectType": {},
"f:filter": {
".": {},
"f:namespaceFilter": {
".": {},
"f:includeNamespaces": {}
}
},
"f:includeOptionalResourceTypes": {},
"f:includeResources": {},
"f:isSFR": {},
"f:keepCrStatus": {},
"f:namespaceMapping": {
".": {},
"f:cm-ns": {},
"f:secret-ns": {}
},
"f:rancherProjectMapping": {},
"f:replacePolicy": {},
"f:storageClassMapping": {},
"f:virtualMachineFiles": {},
"f:virtualMachineName": {},
"f:vmRestoreOptions": {
".": {},
"f:skipMacMasking": {},
"f:skipVmRestart": {}
}
},
"f:status": {
".": {},
"f:finishTimestamp": {},
"f:largeResourceEnabled": {},
"f:resourceCount": {},
"f:resourcerestorestate": {},
"f:resources": {},
"f:restoredresourceCount": {},
"f:singleFileRestoreStatus": {
".": {},
"f:failedFiles": {},
"f:files": {},
"f:finishTimestamp": {},
"f:lastUpdateTimestamp": {},
"f:reason": {},
"f:status": {},
"f:totalFiles": {}
},
"f:totalSize": {}
}
}
},
{
"manager": "stork",
"operation": "Update",
"apiVersion": "stork.libopenstorage.org/v1alpha1",
"time": "2026-01-28T03:59:29Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:finalizers": {
".": {},
"v:\"stork.libopenstorage.org/finalizer-cleanup\"": {}
}
},
"f:status": {
"f:lastUpdateTimestamp": {},
"f:reason": {},
"f:stage": {},
"f:status": {},
"f:volumes": {}
}
}
}
]
},
"spec": {
"backupName": "bkp-cm-secret-nfs-85d1556",
"backupLocation": "test-nfs-bl-85d1556",
"namespaceMapping": {
"cm-ns": "n2-cm",
"secret-ns": "n2-secret"
},
"replacePolicy": "Delete",
"includeOptionalResourceTypes": null,
"includeResources": null,
"storageClassMapping": null,
"rancherProjectMapping": null,
"backupObjectType": "All",
"keepCrStatus": false,
"vmRestoreOptions": {
"skipMacMasking": false,
"skipVmRestart": false
},
"filter": {
"namespaceFilter": {
"includeNamespaces": [
"cm-ns",
"secret-ns"
]
}
},
"isSFR": false,
"virtualMachineName": "",
"virtualMachineFiles": null
},
"status": {
"stage": "Applications",
"status": "InProgress",
"reason": "Application resources restore is in progress",
"resources": [
{
"name": "test86",
"namespace": "n2-cm",
"group": "",
"version": "v1",
"kind": "ConfigMap",
"status": "Successful",
"reason": "Resource restored successfully"
},
{
"name": "test87",
"namespace": "n2-cm",
"group": "",
"version": "v1",
"kind": "ConfigMap",
"status": "Successful",
"reason": "Resource restored successfully"
},
{
"name": "test88",
"namespace": "n2-cm",
"group": "",
"version": "v1",
"kind": "ConfigMap",
"status": "Successful",
"reason": "Resource restored successfully"
},
{
"name": "test89",
"namespace": "n2-cm",
"group": "",
"version": "v1",
"kind": "ConfigMap",
"status": "Failed",
"reason": "Error applying resource: STUB: simulated failure for resource test89 as configured in stubResourcesToFail"
},
{
"name": "test-secret30",
"namespace": "n2-secret",
"group": "",
"version": "v1",
"kind": "Secret",
"status": "Failed",
"reason": "Error applying resource: STUB: simulated failure for resource test-secret30 as configured in stubResourcesToFail"
},
{
"name": "test-secret9",
"namespace": "n2-secret",
"group": "",
"version": "v1",
"kind": "Secret",
"status": "Failed",
"reason": "Error applying resource: STUB: simulated failure for resource test-secret9 as configured in stubResourcesToFail"
}
],
"volumes": [],
"finishTimestamp": null,
"lastUpdateTimestamp": "2026-01-28T03:59:29Z",
"totalSize": 0,
"resourceCount": 0,
"largeResourceEnabled": false,
"restoredresourceCount": 0,
"resourcerestorestate": "",
"singleFileRestoreStatus": {
"status": "",
"reason": "",
"files": null,
"finishTimestamp": null,
"lastUpdateTimestamp": null,
"totalFiles": 0,
"failedFiles": 0
}
}
}
Metadata fields
| Field | Description | Example |
|---|---|---|
metadata.name | Name of the restore custom resource | n2-85d1556 |
metadata.namespace | Kubernetes namespace where the restore CR exists | kube-system |
metadata.uid | Unique identifier assigned by Kubernetes to the restore object | 898XXXXX-abcd-cbcd-8b16-f03a59c0b77f |
metadata.resourceVersion | Internal Kubernetes version used for object change tracking | 329877 |
metadata.generation | Number of times the desired spec has changed | 6 |
metadata.creationTimestamp | Timestamp when the restore CR was created | 2026-01-28T03:58:59Z |
metadata.labels | Labels associated with the restore CR | app.kubernetes.io/part-of: px-backup |
metadata.annotations | Additional metadata used by Portworx Backup for restore context | Cluster name, restore name, organization ID |
Restore specification fields
| Field | Description | Example |
|---|---|---|
spec.backupName | Name of the backup used for the restore | bkp-cm-secret-nfs-85d1556 |
spec.backupLocation | Backup location from which the restore is performed | test-nfs-bl-85d1556 |
spec.namespaceMapping | Mapping of source namespaces to destination namespaces | cm-ns → n2-cm, secret-ns → n2-secret |
spec.replacePolicy | Defines how existing resources are handled during restore | Delete |
spec.includeOptionalResourceTypes | Optional resource types included in restore | null |
spec.includeResources | Specific resources selected for restore | null |
spec.storageClassMapping | Mapping of source storage classes to destination storage classes | null |
spec.rancherProjectMapping | Rancher project mapping used during restore | null |
spec.backupObjectType | Type of backup object being restored | All |
spec.keepCrStatus | Indicates whether restore CR status should be retained | false |
spec.filter.namespaceFilter.includeNamespaces | Namespaces selected for restore | cm-ns, secret-ns |
spec.vmRestoreOptions.skipMacMasking | Skips MAC address masking during VM restore | false |
spec.vmRestoreOptions.skipVmRestart | Skips VM restart after restore | false |
spec.virtualMachineName | Name of VM being restored if VM restore is used | "" |
spec.virtualMachineFiles | File paths used for VM file-level restore | null |
spec.isSFR | Indicates whether this is a single-file restore operation | false |
Restore status fields
| Field | Description | Example |
|---|---|---|
status.stage | Current stage of the restore process | Applications |
status.status | Overall restore status | InProgress |
status.reason | Description of current restore state | Application resources restore is in progress |
status.lastUpdateTimestamp | Timestamp of the most recent status update | 2026-01-28T03:59:29Z |
status.finishTimestamp | Timestamp when restore completed | null |
status.resourceCount | Total number of resources processed | 0 |
status.restoredresourceCount | Number of resources successfully restored | 0 |
status.totalSize | Total size of restored resources | 0 |
status.largeResourceEnabled | Indicates whether the restore was treated as a large resource restore | false |
Individual resource restore status (status.resources[]) fields
| Field | Description | Example |
|---|---|---|
name | Name of the Kubernetes resource | test89 |
namespace | Namespace of the resource | n2-cm |
group | API group of the resource | "" |
version | API version of the resource | v1 |
kind | Kubernetes resource type | ConfigMap |
status | Restore status of the resource | Successful / Failed |
reason | Detailed result or error message for the restore operation | Resource restored successfully |
Single file restore status fields
| Field | Description | Example |
|---|---|---|
status.singleFileRestoreStatus.status | Status of file-level restore operation | "" |
status.singleFileRestoreStatus.reason | Reason for success or failure | "" |
status.singleFileRestoreStatus.totalFiles | Number of files processed in restore | 0 |
status.singleFileRestoreStatus.failedFiles | Number of files that failed during restore | 0 |