Rotate the MongoDB Master Encryption Key

Applicable to both Classic and Federated modes Portworx Backup 3.2.0 and later

Starting with Portworx Backup 3.2.0, you can replace the MongoDB master encryption key as part of a regular Helm upgrade. Provide the new key in the pxc-credentials secret and run the upgrade; Portworx Backup switches to the new key while the upgrade runs. Data that was encrypted before the change remains readable afterward.

note

• The key is replaced by a Helm upgrade. Portworx Backup does not support on-demand or scheduled rotation, and there is no separate command to run. For the upgrades that can carry a key change, see Which upgrades can replace the key.
• The new key must be exactly 96 characters, the same requirement that applies at installation time.
• Existing encrypted data is not rewritten, so the operation completes quickly regardless of how much data or how many backups you have.
• Supplying the same key again makes no change, so it is safe to repeat an upgrade.
• Rotating the underlying data encryption key (re-encrypting the stored data) is not supported.

Which upgrades can replace the key

The Helm upgrade replaces the key when the target Portworx Backup version is 3.2.0 or later. The following upgrade scenarios are supported:

Upgrade	Example	Key replaced?
From an earlier release to 3.2.0 or later	3.1.1 to 3.2.0	Yes
Between releases 3.2.0 and later	3.2.0 to 3.2.x	Yes
Helm upgrade at the same version	3.2.0 to 3.2.0	Yes
To 3.1.x or earlier	3.0.0 to 3.1.1	No

To replace the key without changing the Portworx Backup version, run the Helm upgrade for the currently installed version after changing only the key in pxc-credentials.

For upgrades to Portworx Backup 3.1.x or earlier, a new key in pxc-credentials is ignored and the existing key remains in use.

Prerequisites

• MongoDB encryption is enabled. This procedure replaces an existing master encryption key; it does not enable encryption on a deployment where encryption is disabled. To verify that encryption is enabled, check the value of pxbackup.encryptMongoDB in the Helm release:

  helm get values px-central -n <pxb-namespace> -a | grep encryptMongoDB

  The output must show encryptMongoDB: true. If the value is false, there is no master encryption key to replace and this procedure does not apply.

• Back up the Portworx Backup control plane, including the MongoDB persistent volumes, before replacing the key. Replacing the key does not rewrite existing encrypted data, and if the key replacement fails, the existing key remains in use. A current backup provides recovery protection if other issues occur during the upgrade.

• Store the current and new keys securely in your organization's secret manager or key vault. Retain the current key in case you need to revert to it. If the key currently in use is lost, the encrypted data cannot be recovered.

Rotate the key

1. Generate the new key into a file that only you can read, and verify its length:

   umask 077
   openssl rand -hex 48 > new-master-key.txt
   
   # Must print 96
   awk '{ printf "%d\n", length($0) }' new-master-key.txt

2. Store the new key in your organization's secret manager or key vault before you continue, alongside the key that is currently in use.

3. Update the mongodb-master-encryption-key field of the pxc-credentials secret with the new key:

   # Build the patch in a file. Using stringData means no base64 encoding is
   # needed, and the key is never passed as a command-line argument, so it does
   # not appear in your shell history or in the process list.
   printf '{"stringData":{"mongodb-master-encryption-key":"%s"}}' \
     "$(cat new-master-key.txt)" > patch.json
   
   kubectl patch secret pxc-credentials -n <pxb-namespace> \
     --type merge --patch-file patch.json

   Delete the temporary files once the key is safely stored:

   rm -f patch.json new-master-key.txt

   note

   The stringData field is write-only: Kubernetes base64-encodes its contents into data and does not store stringData itself. This avoids the base64 command, whose options differ between Linux and macOS.

   If you manage Portworx Backup with Argo CD

   How you update pxc-credentials depends on how the Secret is managed:

   • If you created pxc-credentials directly on the cluster, as described in Install Portworx Backup using Argo CD, Argo CD does not manage the Secret. Use the kubectl patch command described above to update the key.
   • If pxc-credentials is defined in your Git repository or managed through a sealed or external Secret, update the key at its source and let Argo CD synchronize the change. Do not patch the live Secret directly because Argo CD can revert the change during the next synchronization.

4. Run a Helm upgrade as described in Upgrade Portworx Backup, or sync the application if you use Argo CD. The key is replaced as part of this upgrade.

5. Monitor the upgrade:

   kubectl get pods -n <pxb-namespace> -w
   kubectl get jobs -n <pxb-namespace> | grep pre-upgrade

6. After the upgrade completes, confirm the outcome in the health check report:

   kubectl get cm px-central-report -n <pxb-namespace> -o go-template='{{ index .data "report-<release_version>" }}' | grep -i "master encryption key"

   Replace <release_version> with your Portworx Backup release version (for example, 3.2.0). The report contains one of the following messages:

   Message	Meaning
   MongoDB master encryption key rotated successfully	The key was replaced with the one you supplied.
   MongoDB master encryption key rotation not required; key unchanged	The key you supplied is the same as the one already in use, so no change was made.
   MongoDB master encryption key rotation failed, followed by the reason	The key was not replaced. The previous key remains in use and your data is unaffected. See Troubleshooting.

7. Log in to the Portworx Backup web console and verify that your clusters are in the Online state.

Return to a previous key

To return to a previously used key, set that key in the pxc-credentials Secret and run the Helm upgrade by following the Rotate the key procedure.

caution

To replace the key, modify only the pxc-credentials Secret. Do not manually modify the mongodb-masterkey field in the pxc-backup-mongodb Secret. This field contains the master key that Portworx Backup is actively using. Manually changing it can cause the stored key to no longer match the key that protects the encrypted data, making the data inaccessible and preventing subsequent key replacement.

Troubleshooting key rotation

• The upgrade fails with encryption key has an invalid length. Expected length is 96 bytes

  The key you supplied is not exactly 96 characters. Nothing has changed at this point. Correct the key in pxc-credentials and run the upgrade again.

• The upgrade fails with a message ending in refusing to rewrap

  The full message is:

  DEK is wrapped with neither the current nor the new master key; refusing to rewrap

  Portworx Backup compares the following keys with the key that protects the encrypted data:

  • the active key stored in the mongodb-masterkey field of the pxc-backup-mongodb Secret
  • the new key supplied in the mongodb-master-encryption-key field of the pxc-credentials Secret

  This error occurs when neither key matches the key used to protect the encrypted data. Portworx Backup cannot safely rewrap the data encryption key (DEK) and stops the operation without replacing the master key. This condition can occur if pxc-backup-mongodb was manually modified and no longer contains the key that is actually in use.

  To resolve the issue, set the mongodb-master-encryption-key field in pxc-credentials to the master key that was in use before the mismatch occurred, and run the upgrade again. Portworx Backup validates the supplied key against the encrypted data before proceeding.

  The failed operation does not replace the existing master key. Do not attempt to determine the correct key by trying different key values. If the master key that protects the encrypted data is no longer available, the encrypted data cannot be recovered.

  If you do not know the correct key, collect the following information and contact Everpure Support:

  # Full health check report, including the failure message
  kubectl describe cm px-central-report -n <pxb-namespace>
  
  # Pre-upgrade job logs
  kubectl logs job/pre-upgrade-check -n <pxb-namespace>
  
  # Status of the pods in the Portworx Backup namespace
  kubectl get pods -n <pxb-namespace>
  
  # Portworx Backup version and Helm release history
  helm list -n <pxb-namespace>
  helm history px-central -n <pxb-namespace>

  Also note when the key was last changed and whether any secret in the namespace was edited manually.

  caution

  Do not share key values or the complete contents of the pxc-credentials and pxc-backup-mongodb Secrets through email or other unsecured channels. If this information is required for troubleshooting, Everpure Support will provide instructions for sharing it securely.

• The health check report says that rotation failed

  The previous key remains in use when rotation fails, so there is no data loss. Collect the report and the pre-upgrade job logs and contact Everpure Support:

  kubectl describe cm px-central-report -n <pxb-namespace>
  kubectl logs job/pre-upgrade-check -n <pxb-namespace>

  The pre-upgrade job is removed when it succeeds, so these logs are available only when it fails.

• Portworx Backup does not start after the upgrade

  The new key is put into use only after Portworx Backup verifies that it works, so this is usually an ordinary pod issue rather than a key issue. Check the pod events and logs as you would for any other upgrade:

  kubectl describe pod -l app=px-backup -n <pxb-namespace>
  kubectl logs deploy/px-backup -n <pxb-namespace>

  If the key was replaced but the px-backup pod was not restarted, restart the deployment so that it picks up the new key:

  kubectl rollout restart deployment px-backup -n <pxb-namespace>