Delete NFS Backups with Job Limit

Overview

When deleting backups that reside on an NFS backup location, Portworx Backup initiates a cleanup process to delete backup data from the storage backend. For each PVC Portworx Backup spawns a separate delete job. These jobs delete the files and directories associated with the volumes. Previously, there was no upper limit on the number of delete jobs that can run concurrently. This behavior allowed for rapid cleanup by leveraging parallel deletion mechanism specifically when a large number of backups are deleted at once.

However, in application clusters with strict job quotas (such as clusters that enforce a limit on the number of concurrent jobs), this parallel deletion approach can overwhelm system resources, potentially leading to performance issues or job failures due to exceeding the allowed job limit.

To address this, Portworx Backup now introduces a configurable concurrency limit for delete jobs. This enhancement enables users to:

• Control how many delete jobs run in parallel during backup deletion

• Prevent cluster overload and adhere to quota restrictions

This topic provides instructions on applying the new concurrency limit to manage deletion operations in constrained environments.

Prerequisites

• Access to Portworx Backup web console

• Privileges to delete backups

• NFS backups configured in the system

• Portworx Backup 2.9.1 and later

Parameters

px-backup-config parameters

The following parameters are introduced as part of Portworx Backup version 2.9.1 and are part of px-backup-config ConfigMap. This ConfigMap resides in the namespace where Portworx Backup is deployed (often referred to as <pxb-namespace>) on Portworx Backup cluster.

Parameter	Description	Default Value	Lower Limit	Upper Limit
NFS_DELETE_JOB_LIMIT	Limits the number of job pods spawned to delete Portworx volumes during backup deletion.	25	1	Based on cluster pod quota limit
NFS_CSI_DELETE_JOB_LIMIT	Limits the number of job pods created to delete CSI volumes during backup deletion.	25	1	Based on cluster pod quota limit
KDMP_DELETE_JOB_LIMIT	Limits the number of job pods spawned to delete CSI + Offload volumes during backup deletion. This is used for [CSI+Offload](link to backup topic).	5	1	Based on cluster pod quota limit

How it works

Let us consider a scenario where you select two Portworx volume backups that use NFS backend storage for deletion through the Portworx Backup. Suppose these backups collectively have 100 PVCs.

If you configure NFS_DELETE_JOB_LIMIT: "50", Portworx Backup will initiate the deletion process by launching 50 delete jobs, each responsible for cleaning up one PVC.

The deletion workflow proceeds in these stages:

1. 50 PVCs are deleted in parallel. The remaining 50 PVCs are queued.

2. Once PVC(s) are deleted out of 50, the subsequent PVCs are picked up for deletion. At any point throughout the deletion process, the number of deletion job pods remain 50.

This continues until all 100 PVCs are deleted. This throttled approach ensures that the number of concurrent delete pods does not exceed the configured limit of 50, preventing resource exhaustion on clusters with job quotas.

The deletion time is primarily determined by the number and size of PVCs and the configured limit contained within the selected backups, not by the total number of backups chosen for deletion.

For more information on states of deletion and how to delete a backup, refer to Delete backup.

note

Resource deletion is performed as a single operation. All resources associated with a backup are deleted by a single pod.

How to configure

Configure px-backup-config parameters

The px-backup-config parameters are hidden by default and do not appear in the ConfigMap. If you don't manually add these parameters with your own values, they are still applied in the background. In such cases, Portworx Backup uses the default values for those parameters. To manually edit the ConfigMap and append the values, perform the following steps:

1. Edit the px-backup-config ConfigMap:

   kubectl edit cm px-backup-config -n <pxb-namespace>

   Sample px-backup-config before appending the parameters:

   apiVersion: v1
   data:
     ALL-CLUSTER-UPDATED-TO-BETA1: "true"
     BACKUP_OBJECTS_MIGRATION_SUCCESS: "true"
     BACKUP_SYNC_LIMIT_CPU: ""
     BACKUP_SYNC_LIMIT_MEMORNFS: ""
     BACKUP_SYNC_REQUEST_CPU: ""
     BACKUP_SYNC_REQUEST_MEMORY: ""
     KDMP_KOPIAEXECUTOR_IMAGE: portworx/kopiaexecutor:<kopia-version>
     KDMP_KOPIAEXECUTOR_IMAGE_SECRET: ""
     OBJECT-LOCK-INCR-BACKUP-COUNT: ""
     OBJECT-LOCK-RETAIN-COUNT: ""
     RESTORE_OBJECTS_MIGRATION_SUCCESS: "true"
   kind: ConfigMap
   metadata:
     creationTimestamp: "2025-08-04T09:02:52Z"
     name: px-backup-config
     namespace: px-backup
     resourceVersion: "<resource-version>"
     uid: <uid>

2. Add the following parameters under the data section in the key-value pair in the below format:

   NFS_DELETE_JOB_LIMIT: "25"
   NFS_CSI_DELETE_JOB_LIMIT: "25"
   KDMP_DELETE_JOB_LIMIT: "5"

   Note that the vales assigned here are default values, you need to provide values based on your cluster requirements.

3. Save and exit the editor.

4. Verify that the new values updated in the ConfigMap are reflected in the required clusters:

   kubectl get cm px-backup-config -n <pxb-namespace> -o yaml

   Example

   Here is a sample snippet from the px-backup-config ConfigMap after adding the parameters:

   apiVersion: v1
   data:
     ALL-CLUSTER-UPDATED-TO-BETA1: "true"
     BACKUP_OBJECTS_MIGRATION_SUCCESS: "true"
     BACKUP_SYNC_LIMIT_CPU: ""
     BACKUP_SYNC_LIMIT_MEMORNFS: ""
     BACKUP_SYNC_REQUEST_CPU: ""
     BACKUP_SYNC_REQUEST_MEMORY: ""
     KDMP_KOPIAEXECUTOR_IMAGE: portworx/kopiaexecutor:<kopia-version>
     KDMP_KOPIAEXECUTOR_IMAGE_SECRET: ""
     NFS_CSI_DELETE_JOB_LIMIT: "50"
     NFS_DELETE_JOB_LIMIT: "50"
     KDMP_DELETE_JOB_LIMIT: "50"
     OBJECT-LOCK-INCR-BACKUP-COUNT: ""
     OBJECT-LOCK-RETAIN-COUNT: ""
     RESTORE_OBJECTS_MIGRATION_SUCCESS: "true"
   kind: ConfigMap
   metadata:
     creationTimestamp: "2025-08-04T09:02:52Z"
     name: px-backup-config
     namespace: px-backup
     resourceVersion: "<resource-version>"
     uid: <resource-uid>

Best Practices

• Start with conservative limits and increase gradually only if your cluster has sufficient resources.

• Monitor job behavior and system load during large-scale deletions.

• Coordinate with your platform or DevOps team if your cluster has a predefined Job quota (for example, 250 delete jobs).

Related Topics

• NFS backup