Password Policy for Upgrading to Portworx Backup 2.10.0

Portworx Backup provides a streamlined mechanism using Kubernetes secrets to manage and rotate credentials for its internal databases. You can also enable optional encryption of the internal database and specify encryption keys via Kubernetes secrets, enhancing Portworx Backup's security and protection capabilities. For more details, refer to the Password Policy.

When upgrading from an earlier release to Portworx Backup 2.10.0, you have two password policy choices:

Option 1: Upgrade with New Custom Passwords (Recommended)

This option overrides any existing passwords with new custom passwords that comply with the updated security policy. This approach is recommended by Portworx for enhanced security.

Prerequisites: Check the health status of pods in your Portworx Backup control plane before proceeding.

• For Healthy Pods: If all pods in the Portworx Backup namespace are running and healthy, follow the steps in Upgrade with Security-Compliant Custom Passwords.

• For Unhealthy Pods: If any pods are in an unhealthy state, follow the steps in Upgrading with Custom Password (Unhealthy Pods).

Upgrade with Security-Compliant Custom Passwords

1. Pre-upgrade steps

   • Create pxc-credentials as per the password policy
   • Get existing installation values and helm update

     helm get values --namespace <pxb-namespace> px-central -o yaml > values-current.yaml
     helm repo add portworx http://charts.portworx.io/ && helm repo update

2. Upgrade

   helm upgrade px-central portworx/px-central --namespace central --create-namespace --version 2.10.0 -f values-current.yaml

3. Monitor upgrade

   # Watch upgrade progress
   kubectl get pods -n <px-backup-namespace> -w
   
   # Check pre-upgrade hook status
   kubectl get jobs -n <px-backup-namespace> | grep pre-upgrade
   
   # Monitor pre-upgrade hook logs
   kubectl logs job/pre-upgrade-check -n <px-backup-namespace>

4. Monitor the pods and log into web console.

   # Monitor the pods
   kubectl get pods -n px-backup

Upgrading with Custom Password (Unhealthy Pods)

This scenarios covers upgrading Portworx backup with customised password when any of the pods in Portworx Backup namespace are not in healthy state.

1. Identify Unhealthy Pods

   # Check pod status in PX-Backup namespace
   kubectl get pods -n <px-backup-namespace>
   
   # Check MongoDB StatefulSet specifically
   kubectl get statefulset pxc-backup-mongodb -n <px-backup-namespace>
   
   # Check pod logs for errors
   kubectl logs pxc-backup-mongodb-0 -n <px-backup-namespace>

   Fix the current issues, especially mongodb pods.

2. Make sure to create pxc-credentials secret before starting the upgrade process.

3. Verify Current Password Configuration

   # Check existing MongoDB secret
   kubectl get secret pxc-backup-mongodb -n <px-backup-namespace> -o yaml
   
   # Decode current passwords (if needed for reference)
   kubectl get secret pxc-backup-mongodb -n <px-backup-namespace> \
   -o jsonpath='{.data.mongodb-root-password}' | base64 -d

4. Increase the timeout value(--timeout=60m) to have enough time to finish the upgrade.

Upgrade procedure

1. Create pxc-credential secret as per the password policy.

2. Current values of Portworx backup installation and modify the values as per the need.

   helm get values --namespace <pxb-namespace> px-central -o yaml > values-current.yaml

3. Upgrade

   helm upgrade px-central portworx/px-central --namespace <pxb-namespace> --version 2.10.0 -f values-px-central.yaml --timeout=60m

4. Monitor Upgrade Progress

    # Watch upgrade progress
    kubectl get pods -n <px-backup-namespace> -w
   
    # Check pre-upgrade hook status
    kubectl get jobs -n <px-backup-namespace> | grep pre-upgrade
   
    # Monitor pre-upgrade hook logs
    kubectl logs job/pre-upgrade-check -n <px-backup-namespace>

Troubleshooting

1. Pre-upgrade hook fails due to unhealthy pods If a pre-upgrade hook fails due to unhealthy pods, it's recommended to add skip validations flag to bypass the checks however mongodb pods should be healthy to proceed with upgrade.

   # Solution: Skip validations
   helm upgrade px-backup portworx/px-central --set pxbackup.skipValidations=true 

2. MongoDB authentication fails after upgrade If mongodb authentication fails after upgrade, then use the following commands to validate the updated mongodb passwords.

   kubectl exec -it pxc-backup-mongodb-0 -n <px-backup-namespace> -- \
   mongosh admin -u root -p '<your-root-password>' --eval "print('Root password is valid!');"

   kubectl exec -it pxc-backup-mongodb-0 -n <px-backup-namespace> -- \
   mongosh px-backup -u px-backup -p '<your-custom-password>' --eval "print('PX-Backup password is valid!');"

3. Check Upgrade hook logs for the more debugging information.
4. During upgrade, if mongodb passwords were updated but it failed later due to some timeout. Then its recommended to restart the upgrade again.

Option 2: Upgrade While Retaining Existing Passwords

This option preserves your current passwords from the older version during the upgrade process. Follow the steps in Preserving Existing Passwords During Upgrade to maintain password continuity.

Preserving Existing Passwords During Upgrade

If you are upgrading an existing Portworx Backup installation that uses custom passwords, you want to retain those passwords during the upgrade. To do so, follow these steps:

1. Extract Current Passwords Before upgrading, extract and save current passwords:

   # Create backup of current passwords
   mkdir -p px-backup-password-backup
   cd px-backup-password-backup
   
   # Extract MongoDB passwords
   kubectl get secret pxc-backup-mongodb -n <px-backup-namespace> \
       -o jsonpath='{.data.mongodb-root-password}' | base64 -d > mongodb-root-password.txt
   
   kubectl get secret pxc-backup-mongodb -n <px-backup-namespace> \
       -o jsonpath='{.data.mongodb-password}' | base64 -d > mongodb-px-backup-password.txt
   
   kubectl get secret pxc-backup-mongodb -n <px-backup-namespace> \
       -o jsonpath='{.data.mongodb-replica-set-key}' | base64 -d > mongodb-replica-set-key.txt
   
   # Extract other passwords
   kubectl get secret pxcentral-keycloak-postgresql -n <px-backup-namespace> \
       -o jsonpath='{.data.postgresql-password}' | base64 -d > postgresql-password.txt
   
   kubectl get secret pxcentral-mysql-secret -n <px-backup-namespace> \
       -o jsonpath='{.data.DB_PASSWORD}' | base64 -d > mysql-password.txt

2. Create Retention Secret

   Create a secret with existing passwords to ensure continuity.

   # Read extracted passwords
   MONGODB_ROOT_PASSWORD=$(cat mongodb-root-password.txt)
   MONGODB_PX_BACKUP_PASSWORD=$(cat mongodb-px-backup-password.txt)
   MONGODB_REPLICA_SET_KEY=$(cat mongodb-replica-set-key.txt)
   POSTGRESQL_PASSWORD=$(cat postgresql-password.txt)
   MYSQL_PASSWORD=$(cat mysql-password.txt)
   
   # Create retention secret
   kubectl create secret generic pxc-credentials \
   --namespace=<px-backup-namespace> \
   --from-literal=mongodb-px-backup-password="$MONGODB_PX_BACKUP_PASSWORD" \
   --from-literal=mongodb-root-password="$MONGODB_ROOT_PASSWORD" \
   --from-literal=mongodb-replica-set-key="$MONGODB_REPLICA_SET_KEY" \
   --from-literal=postgresql-password="$POSTGRESQL_PASSWORD" \
   --from-literal=mysql-password="$MYSQL_PASSWORD" \
   --dry-run=client -o yaml | kubectl apply -f -

3. Apply the secret before upgrade procedure, and after upgrade ensure that the passwords are retained.