Version: Next

Restore Generic/KDMP backups

KDMP (Kubernetes Data Management Platform) ensures that Kubernetes applications are backed up in a consistent state, including both stateless and stateful applications. Portworx Backup utilizes the KDMP driver to attach the mount point of the PV on the worker node and copies the file system contents to object storage. KDMP restore functionality allows users to recover Kubernetes workloads, including data and application configurations from backups.

The KDMP restore operation fails when the snapshot size or the contents of the Persistent Volume Claim (PVC) exceed the size of the PVC, particularly for file system-based storage provisioners. By default, Portworx Backup increases the restore PVC size for KDMP by 10% for the restore to succeed. This functionality pre-allocates additional storage space for specific storage capacity requirements during the restore.

However, if the restore continues to fail due to insufficient space, Portworx Backup records the failure reason in the KDMP job pod logs and Stork logs. If job pod logs are unavailable, you can check the Stork logs. Here is a typical KDMP restore failure error message:

Sample Error message

time="2024-09-05T11:45:19Z" level=error msg="runRestore: restore failed: failed to run the restore command: exit status 1 stdout:  stderr: Restoring to local filesystem (/data) with parallelism=8...\nProcessed 102 (1 GB) of 101 (1 GB) 32.3 MB/s (100.0%) remaining 0s.\nProcessed 102 (1 GB) of 101 (1 GB) 32.3 MB/s (100.0%) remaining 0s.\n\x1b[31mERROR\x1b[0m error restoring: restore error: copy file: error creating file: cannot write data to file %q /data/sample1: write /data/sample1: no space left on device\n"
error: restore failed: failed to run the restore command: exit status 1 stdout:  stderr: Restoring to local filesystem (/data) with parallelism=8...
Processed 102 (1 GB) of 101 (1 GB) 32.3 MB/s (100.0%!)(MISSING) remaining 0s.
Processed 102 (1 GB) of 101 (1 GB) 32.3 MB/s (100.0%!)(MISSING) remaining 0s.
ERROR error restoring: restore error: copy file: error creating file: cannot write data to file %!q(MISSING) /data/sample1: write /data/sample1: no space left on device

Resolution

To resolve this issue, Portworx Backup has introduced a new environment variable called KDMP_RESTORE_PVC_SIZE_PERCENTAGE in kdmp-config ConfigMap. This environment variable allows you to specify the percentage increase (or decrease) in size of restore PVC during restore process.

For example, if backed up PVC size is 10Gi, during restore if you want to increase the restore PVC size to 15% more than that of backed up PVC, you need to set the value of this environment variable to 15. With this, the size of the restore PVC increases to 11.5Gi [10+1.5(15% increase) = 11.5Gi]

To achieve this you need to edit the kdmp-config ConfigMap and increase the restore PVC size by modifying the KDMP_RESTORE_PVC_SIZE_PERCENTAGE parameter to match the snapshot size.

To change the default value of KDMP_RESTORE_PVC_SIZE_PERCENTAGE environment variable:

Execute the following command to edit the kdmp-config from your application cluster:

kubectl edit cm kdmp-config -n kube-system

Update the value of KDMP_RESTORE_PVC_SIZE_PERCENTAGE with the required value:

apiVersion: v1
data:
  KDMP_BACKUP_JOB_LIMIT: "5"
  KDMP_DELETE_JOB_LIMIT: "5"
  KDMP_KOPIAEXECUTOR_IMAGE: kopiaexecutor:1.2.14
  KDMP_KOPIAEXECUTOR_IMAGE_SECRET: ""
  KDMP_KOPIAEXECUTOR_LIMIT_CPU: "0.2"
  KDMP_KOPIAEXECUTOR_LIMIT_MEMORY: 1Gi
  KDMP_KOPIAEXECUTOR_REQUEST_CPU: "0.1"
  KDMP_KOPIAEXECUTOR_REQUEST_MEMORY: 700Mi
  KDMP_MAINTENANCE_JOB_LIMIT: "5"
  KDMP_RESTORE_JOB_LIMIT: "5"
  KDMP_RESTORE_PVC_SIZE_PERCENTAGE: "15"
  SNAPSHOT_TIMEOUT: ""
kind: ConfigMap
metadata:
  creationTimestamp: "2024-06-13T10:07:25Z"
  name: kdmp-config
  namespace: kube-system
  resourceVersion: "<resource-version>"
  uid: <resource-uid>

In the above example, the restore PVC size will be 15% more than the original PVC (PVC that was backed up) size.

Related topics