Health Check for Install and Upgrade

Applicable to both Classic and Federated modes

Introduction

Portworx Backup includes a health check feature to streamline installation and upgrades while reducing failures and improving user experience. This check is designed to proactively validate environments and detect potential issues to help users in troubleshooting the issues that may appear during Portworx Backup install or upgrade. Health check is a job that runs before install or upgrade and aims to:

• Streamline the installation and upgrade experience
• Reduce failure rates through early detection and fail-fast mechanisms
• Simplify debugging with clear insights and recommendations

The health check feature is available for environments that are currently supported by Portworx Backup. Also, health check automatically deletes the post-install hook job and eliminates the process of manual deletion during upgrade.

caution

Health check validations are enabled by default during Portworx Backup installation or upgrade. You can disable them for testing purposes by appending the following string in the Portworx Central specgen during install or upgrade:

--set pxbackup.skipValidations=true

Setting skipValidations=true bypasses all health check validations described on this page, not just a single check. This includes the Persistent Volume Claim cleanup validation, which guards against the data conflict or data-loss scenario caused by stale PVCs from a previous installation. With validations skipped, an install or upgrade can proceed in an unsupported or unhealthy state and may fail later or leave the deployment in an inconsistent state. Use this flag only in non-production or testing environments.

Review the health check report after every install or upgrade

The health check stores all warnings and failures in a ConfigMap report, which it generates for every installation and upgrade. Because a successful operation can still produce warnings, always review the report after the operation completes, even if the installation or upgrade succeeds. To retrieve the report and map its entries to the checks below, run:

kubectl get cm px-central-report -n <pxb-namespace> -o go-template='{{ index .data "report-<release_version>" }}'

For details, see Retrieving health check results after a failed install or upgrade.

Key validations

1. Kubernetes version validation

• Ensures compatibility by checking Kubernetes cluster versions before installation or upgrades.

• Provides warnings if the cluster is running an unsupported version.

2. Namespace quota check for PVCs

• Validates the resource quota for Persistent Volume Claims (PVCs) of the namespace where Portworx Backup will be deployed or upgraded. The check compares the PVC count and storage that Portworx Backup components request against any ResourceQuota configured on the target namespace.

• Prevents deployment or upgrade if the namespace quota is too low to satisfy those PVC storage requirements. To resolve a failure, ask your cluster administrator to raise the PVC count and storage limits in the namespace ResourceQuota (or remove the quota) so that it can accommodate the Portworx Backup PVCs, then retry the install or upgrade.

3. StorageClass validation

• Verifies the existence of the specified storage class.

• Blocks the install/upgrade until the user provides a valid storage class.

4. Persistent Volume Claim cleanup validation

• Checks for leftover PVCs from previous installations.

• Prevents installation/upgrade until stale resources (PVCs) are cleaned up.

  note

  Stale PVCs are Persistent Volume Claims left behind from a previous incomplete or failed installation. They may contain backup metadata and can cause data conflicts or loss during reinstallation. Remove them before retrying the installation.

5. Port information display in Helm notes

• Displays port-related details directly in Helm notes for user convenience.

• Includes links to documentation for detailed network prerequisites.

6. Pre-upgrade health check

• Ensures all Portworx Backup pods are in a healthy state before upgrades.

• Blocks upgrade if pods are not in a healthy state.

caution

If the primary MongoDB pod goes down at any time either before or after the Portworx Backup installation or upgrade, it is essential for the secondary peers to be healthy and DNS-resolvable to maintain a stable replica set.

7. Pre-install/pre-upgrade reports

• Collects and stores all warnings generated during installation or upgrades in the respective pod logs.

• Provides the following easy-to-access command to retrieve the report with warnings:

  kubectl get cm px-central-report -n <pxb-namespace> -o go-template='{{ index .data "report-<release_version>" }}'

  Replace <release_version> with your Portworx Backup release version (for example, 3.1.0).

• Generates a comprehensive pre-install and pre-upgrade report.

8. Unsupported path upgrades

• Blocks upgrades if the installed version is more than two versions (n-2) behind the target release (current release).

9. Validation for file-system-based provisioners

• Prevents installation on storage classes using file-system-based provisioners.

10. Image validation

• Validates image repository and tags during installation and upgrade.

Federated mode validations

Federated mode adds the validations described below. Unlike the install and upgrade health checks listed above, the first two run at runtime — when you perform an operation — not during install or upgrade:

• The Stork Workload Identity validation runs whenever you onboard an application cluster, and the backup location type validation runs whenever you add or validate a backup location. These validations run after Portworx Backup is installed, including when the corresponding operation is performed after an upgrade. They do not run as part of the installation or upgrade process.
• The in-place upgrade block is the only Federated-specific check that runs as part of the install/upgrade health check (during pre-upgrade) on the backup cluster.

Stork Workload Identity (WLI) configuration

Runs at runtime when you onboard an application cluster in Federated mode. It is not performed during the pre-install or pre-upgrade health check.

• What it checks: The live Stork configuration on the application cluster, including the StorageCluster (STC) custom resource, the Stork service account annotations, and the namespace label applied by the Portworx Operator. The validation inspects the cluster's current state rather than information stored on the Portworx Backup server.
• Verifies that Stork is configured with the correct Workload Identity mechanism for the backup location provider:
  • Azure Blob Storage: Azure Managed Identity with the required Managed Identity client ID annotation.
  • AWS S3: AWS IAM Roles for Service Accounts (IRSA) with the required IAM role ARN annotation.
  • Google Cloud Storage (GCS): GCP Workload Identity Federation with the required Workload Identity configuration.
  • Also verifies that the required namespace label is present.
• Severity: Hard block. If the required Workload Identity configuration is missing or incomplete, application cluster onboarding fails and the cluster is not added. To verify the configuration before onboarding, see Verify Stork configuration on each application cluster.

Backup location type validation

Runs at runtime when you add or manually validate a backup location in Federated mode. It is not performed during installation or upgrade. For more information, see Add a backup location and Validate cluster connectivity to a backup location.

• What it checks: The backup location type selected when the backup location is added or validated, evaluated against the list of backup location types supported in Federated mode.
• Verifies that the backup location uses a supported storage provider: Azure Blob Storage, AWS S3, or Google Cloud Storage (GCS).
• In the Portworx Backup web console, the Federated mode backup location creation flow presents only the supported providers, so you cannot select NFS or another unsupported type through the UI. This validation is a safeguard that primarily applies when a backup location is created or validated through the API or CLI.
• Severity: Hard block. If an unsupported storage type (such as NFS) is submitted, validation fails and the backup location is not added.
• This restriction applies only to Federated mode. Classic mode continues to support the full range of backup location types, including S3-compatible object storage, Azure Blob Storage, GCS, and NFS.

For more information, see Configure Backup Locations in Federated Mode.

Block in-place upgrade from Classic mode to Federated mode

Runs during the pre-upgrade health check on the backup cluster.

• Prevents upgrade from an existing Classic mode deployment to Federated mode.
• Federated mode must be installed as a new deployment and cannot be converted from an existing Classic mode deployment.

For a complete overview of all checks and which phase (pre-install or pre-upgrade) they apply to, see the Health Check Matrix.

Pod validation

Portworx Backup validates the following pods for their health during upgrade:

StatefulSets:

• alertmanager-px-backup-alertmanager (only for px-backup version greater than 2.7.0)
• prometheus-px-backup-dashboard-prometheus (only for px-backup version greater than 2.7.0)
• pxc-backup-mongodb
• pxcentral-keycloak
• pxcentral-keycloak-postgresql
• pxcentral-mysql

Deployments:

• px-backup
• pxcentral-apiserver
• pxcentral-backend
• pxcentral-frontend
• pxcentral-lh-middleware
• prometheus-operator (only for px-backup version greater than 2.7.0)

List of unsupported provisioners

Portworx Backup does not support the following cloud and on-premises file system-based provisioners:

Cloud Storage Provisioners

• efs.csi.aws.com – Amazon EFS (Elastic File System) for AWS
• kubernetes.io/azure-file – Azure Files (legacy provisioner)
• file.csi.azure.com – Azure Files CSI driver
• vpc.file.csi.ibm.io – IBM Cloud VPC File Storage
• filestore.csi.storage.gke.io – Google Cloud Filestore

On-Premises Storage Provisioners

• nfs-client – Generic NFS provisioner (can be used for on-premises NFS servers)

• kubernetes.io/glusterfs – GlusterFS (typically used for on-premises storage)

Insecure registry

By default, the health check job uses secure connections (TLS) to validate container images. However, in certain cases, you may need to allow image validation to proceed even when the registry uses self-signed, untrusted, or invalid certificates. To achieve this, set the insecureRegistry flag to true in the values.yaml file during Helm installation.

When to set insecureRegistry to true?

You should enable insecureRegistry in the following scenarios:

• Your container registry uses self-signed certificates.
• You are working with a private, on-premises registry that does not have a valid TLS certificate.
• You are running in an air-gapped environment where external certificate validation is not possible.
• You are testing in a development or staging environment that does not enforce strict TLS security.

How to enable insecure registry mode?

To allow connections to an insecure registry, update the values.yaml file:

insecureRegistry: true

Alternatively, you can enable it during Helm installation using the --set flag in the Portworx Central specgen page:

--set images.insecureRegistry=true

Security considerations

Setting insecureRegistry to true allows connections to registries without verifying TLS certificates. This setting should not be used in production environments unless absolutely necessary. By default, insecureRegistry is set to false, which enforces TLS certificate validation during container image checks. Keep this default unless your registry cannot present a valid TLS certificate.

Next steps

• For detailed upgrade and installation instructions and to learn more about health check, see the following topics:
  • Health check matrix
  • Install
  • Upgrade
• For any issues during install or upgrade, contact Everpure support.