Skip to main content
Version: 3.0

Password Policy for Upgrading to Portworx Backup 2.10.x

Portworx Backup provides a streamlined mechanism using Kubernetes secrets to manage and rotate credentials for its internal databases. You can also enable optional encryption of the internal database and specify encryption keys via Kubernetes secrets, enhancing Portworx Backup's security and protection capabilities. For more details, see the Password Policy.

When upgrading from an earlier release to Portworx Backup 2.10.x, you have two password policy choices:

This option overrides any existing passwords with new custom passwords that comply with the updated security policy. This approach is recommended by Portworx for enhanced security.

Prerequisites: Check the health status of pods in your Portworx Backup control plane before proceeding.

Upgrade with security-compliant custom passwords​

  1. Pre-upgrade steps

    • Create the pxc-credentials secret as per the Password Policy guide.
    • Get existing installation values and update the Helm repo: 
      helm get values --namespace <pxb-namespace> px-central -o yaml > values-current.yaml
      helm repo add portworx https://charts.portworx.io/ && helm repo update

  2. Upgrade

    helm upgrade px-central portworx/px-central --namespace central --create-namespace --version <Variable name="pxbVer_3.0.0"/> -f values-current.yaml

  3. Monitor upgrade

    # Watch upgrade progress
    kubectl get pods -n <px-backup-namespace> -w

    # Check pre-upgrade hook status
    kubectl get jobs -n <px-backup-namespace> | grep pre-upgrade

    # Monitor pre-upgrade hook logs
    kubectl logs job/pre-upgrade-check -n <px-backup-namespace>

  4. Monitor the pods and log into web console.

    # Monitor the pods
    kubectl get pods -n px-backup

Upgrading with custom password (unhealthy pods)​

This scenario covers upgrading Portworx Backup with a custom password when any of the pods in the Portworx Backup namespace are not in a healthy state.

  1. Identify unhealthy pods

    # Check pod status in PX-Backup namespace
    kubectl get pods -n <px-backup-namespace>

    # Check MongoDB StatefulSet specifically
    kubectl get statefulset pxc-backup-mongodb -n <px-backup-namespace>

    # Check pod logs for errors
    kubectl logs pxc-backup-mongodb-0 -n <px-backup-namespace>

    Fix the current issues, especially with MongoDB pods.

  2. Make sure to create the pxc-credentials secret as per the Password Policy guide before starting the upgrade process.

  3. Verify current password configuration

    # Check existing MongoDB secret
    kubectl get secret pxc-backup-mongodb -n <px-backup-namespace> -o yaml

    # Decode current passwords (if needed for reference)
    kubectl get secret pxc-backup-mongodb -n <px-backup-namespace> \
    -o jsonpath='{.data.mongodb-root-password}' | base64 -d

  4. Increase the timeout value (--timeout=60m) to allow enough time to finish the upgrade.

Upgrade procedure​

  1. Create the pxc-credentials secret as per the Password Policy guide.

  2. Retrieve current values of the Portworx Backup installation:

    helm get values --namespace <pxb-namespace> px-central -o yaml > values-current.yaml

  3. Upgrade:

    helm upgrade px-central portworx/px-central --namespace <pxb-namespace> --version <Variable name="pxbVer_3.0.0"/> -f values-current.yaml --timeout=60m

  4. Monitor upgrade progress

     # Watch upgrade progress
     kubectl get pods -n <px-backup-namespace> -w

     # Check pre-upgrade hook status
     kubectl get jobs -n <px-backup-namespace> | grep pre-upgrade

     # Monitor pre-upgrade hook logs
     kubectl logs job/pre-upgrade-check -n <px-backup-namespace>

Troubleshooting​

  1. Pre-upgrade hook fails due to unhealthy pods If a pre-upgrade hook fails due to unhealthy pods, add the skip validations flag to bypass the checks. Note that MongoDB pods must be healthy to proceed with the upgrade. 
    # Solution: Skip validations
    helm upgrade px-central portworx/px-central --set pxbackup.skipValidations=true
  2. MongoDB authentication fails after upgrade If MongoDB authentication fails after the upgrade, use the following commands to validate the updated MongoDB passwords. 
    kubectl exec -it pxc-backup-mongodb-0 -n <px-backup-namespace> -- \
    mongosh admin -u root -p '<your-root-password>' --eval "print('Root password is valid!');"
    kubectl exec -it pxc-backup-mongodb-0 -n <px-backup-namespace> -- \
    mongosh px-backup -u px-backup -p '<your-custom-password>' --eval "print('PX-Backup password is valid!');"
  3. Check upgrade hook logs for additional debugging information.
  4. During upgrade, if MongoDB passwords were updated but the upgrade failed later due to a timeout, restart the upgrade.

Option 2: Upgrade while retaining existing passwords​

This option preserves your current passwords from the older version during the upgrade process. Follow the steps in Preserving Existing Passwords During Upgrade to maintain password continuity.

Preserving existing passwords during upgrade​

If you are upgrading an existing Portworx Backup installation that uses custom passwords, you want to retain those passwords during the upgrade. To do so, follow these steps:

  1. Extract current passwords Before upgrading, extract and save current passwords:

    # Create backup of current passwords
    mkdir -p px-backup-password-backup
    cd px-backup-password-backup

    # Extract MongoDB passwords
    kubectl get secret pxc-backup-mongodb -n <px-backup-namespace> \
        -o jsonpath='{.data.mongodb-root-password}' | base64 -d > mongodb-root-password.txt

    kubectl get secret pxc-backup-mongodb -n <px-backup-namespace> \
        -o jsonpath='{.data.mongodb-password}' | base64 -d > mongodb-px-backup-password.txt

    kubectl get secret pxc-backup-mongodb -n <px-backup-namespace> \
        -o jsonpath='{.data.mongodb-replica-set-key}' | base64 -d > mongodb-replica-set-key.txt

    # Extract other passwords
    kubectl get secret pxcentral-keycloak-postgresql -n <px-backup-namespace> \
        -o jsonpath='{.data.postgresql-password}' | base64 -d > postgresql-password.txt

    kubectl get secret pxcentral-mysql-secret -n <px-backup-namespace> \
        -o jsonpath='{.data.DB_PASSWORD}' | base64 -d > mysql-password.txt

  2. Create retention secret

    Create a secret with the existing passwords to ensure continuity.

    # Read extracted passwords
    MONGODB_ROOT_PASSWORD=$(cat mongodb-root-password.txt)
    MONGODB_PX_BACKUP_PASSWORD=$(cat mongodb-px-backup-password.txt)
    MONGODB_REPLICA_SET_KEY=$(cat mongodb-replica-set-key.txt)
    POSTGRESQL_PASSWORD=$(cat postgresql-password.txt)
    MYSQL_PASSWORD=$(cat mysql-password.txt)

    # Create retention secret
    kubectl create secret generic pxc-credentials \
    --namespace=<px-backup-namespace> \
    --from-literal=mongodb-px-backup-password="$MONGODB_PX_BACKUP_PASSWORD" \
    --from-literal=mongodb-root-password="$MONGODB_ROOT_PASSWORD" \
    --from-literal=mongodb-replica-set-key="$MONGODB_REPLICA_SET_KEY" \
    --from-literal=postgresql-password="$POSTGRESQL_PASSWORD" \
    --from-literal=mysql-password="$MYSQL_PASSWORD" \
    --dry-run=client -o yaml | kubectl apply -f -

  3. Apply the secret to your cluster before running the upgrade. Once applied, proceed with the Helm upgrade command as described in the upgrade guide for your environment:

    After the upgrade completes, verify that the passwords are retained and all pods return to Ready state.

Last updated on