Troubleshooting backup failures with Stork restricted RBAC mode

This section describes how to resolve backup-related failures that can occur when restricted RBAC mode is enabled in Stork.

For information on restricted RBAC mode, see Run Portworx with Restricted RBAC in the Portworx Enterprise documentation.

When restricted RBAC mode is enabled, Stork runs with limited permissions, which can prevent certain backup operations from completing successfully.

Common backup failures

The following table lists common backup failure scenarios and their full error messages that can occur when restricted RBAC mode is enabled in Stork.

note

In this table, PXD (Portworx Data) refers to the native Portworx volume backup type, while KDMP (Kubernetes Data Migration Protocol) refers to the generic snapshot-based backup type.

Scenario	Full error message	Where it appears
PXD backup to NFS fails during resource upload/export	Error uploading resources: [failed in creation of nfs job for resourceexport [mysql-backup/nfs-backup-26de881-mysql-backup], err: failed to get job list: jobs.batch is forbidden: User "system:serviceaccount:portworx:stork" cannot list resource "jobs" in API group "batch" at the cluster scope], namespace: [mysql-backup]	Activity Info
PXD backup fails when pre/post rules are used	Error running PreExecRule: error executing PreExecRule for namespace mysql-backup: could not execute: pods "mysql-7969d55b84-88b77" is forbidden: User "system:serviceaccount:portworx:stork" cannot create resource "pods/exec" in API group "" in the namespace "mysql-backup"	Activity Info
PXD backup to S3 fails when pre/post rules are used	Error running PreExecRule: error executing PreExecRule for namespace mysql-backup: could not execute: pods "mysql-7969d55b84-88b77" is forbidden: User "system:serviceaccount:portworx:stork" cannot create resource "pods/exec" in API group "" in the namespace "mysql-backup"	Activity Info
KDMP backup to NFS fails during backup processing	failed updating DE CR backup-8d1f256-7568a27-mysql-backup: Operation cannot be fulfilled on dataexports.kdmp.portworx.com "backup-8d1f256-7568a27-mysql-backup": the object has been modified; please apply your changes to the latest version and try again failed to delete data export CR [backup-8d1f256-278866f-mysql-backup]: dataexports.kdmp.portworx.com "backup-8d1f256-278866f-mysql-backup" not found failed to delete data export CR [backup-8d1f256-7f3fe39-mysql-backup]: dataexports.kdmp.portworx.com "backup-8d1f256-7f3fe39-mysql-backup" not found	Stork logs
KDMP backup to S3 fails during final stage	Backup failed at stage Final for volume: failed to start a data transfer job, dataexport [backup-f7c9d2c-7f3fe39-mysql-backup]: failed to get job list: jobs.batch is forbidden: User "system:serviceaccount:portworx:stork" cannot list resource "jobs" in API group "batch" in the namespace "mysql-backup"	JSON

note

The exact error message may vary, but these patterns are commonly observed when restricted RBAC mode is enabled in Portworx Backup when RBAC mode is set to restricted in the Portworx Enterprise configuration.

For information on how to check if restricted RBAC mode is enabled in Portworx Enterprise, and how to resolve these issues, see Troubleshooting workflow.

What these errors indicate

These failures typically occur because:

• Stork is running in restricted RBAC mode. For more information, see Run Portworx with Restricted RBAC in the Portworx Enterprise documentation.
• Backup workflows require permissions that are not available in this mode
• As a result, some backup steps (such as job execution or rule execution) fail

Even though the errors may look different, they are all commonly triggered by the same underlying condition of Restricted RBAC mode limiting Stork operations. For more information, see the Common backup failures table above.

Troubleshooting workflow

To identify and resolve the backup-related failures due to Stork restricted RBAC mode, follow these steps:

1. Check if restricted RBAC mode is enabled in Portworx Enterprise.

   kubectl get stc -o yaml

2. Inspect Stork logs

   kubectl logs deployment/stork -n <namespace>

3. Identify RBAC errors by looking for forbidden or cannot messages.

4. Verify ClusterRole

   kubectl get clusterrole <operator-role> -o yaml

   For example,

   kubectl get clusterrole stork -o yaml

5. Test permissions

   kubectl auth can-i <verb> <resource> --as=system:serviceaccount:<ns>:stork

   For example,

   kubectl auth can-i list jobs \
     --as=system:serviceaccount:portworx:stork

How to resolve

Identify these issues using the Troubleshooting workflow, and disable restricted RBAC mode.

Update your StorageCluster (STC) configuration as follows:

spec:
  stork:
    enabled: true
    restrictDataProtectionRBAC: false

For more information on restricted RBAC mode, see Run Portworx with Restricted RBAC in the Portworx Enterprise documentation.