Version: 3.6

Migrate Portworx Datastore from PX-StoreV1 to PX-StoreV2

The Portworx Operator automatically migrates your Portworx datastore from PX-StoreV1 to PX-StoreV2 by using an in-place rolling conversion. The Operator manages the migration, including preflight checks, node conversion, data evacuation, and cluster health monitoring.

How the migration works

The Operator performs the following steps to migrate your cluster from PX-StoreV1 to PX-StoreV2:

Preflight checks: Verify that all nodes meet the PX-StoreV2 requirements. For more information, see System requirements.
Enable mixed mode: Allow PX-StoreV1 and PX-StoreV2 nodes to coexist temporarily during the migration by updating the StorageCluster with PX-StoreV2 settings. The Operator injects allow_mixed_mode=1 into the runtime options, merging it with any existing runtime options you have configured. When the same option is set in multiple places, the Operator applies the following priority order (highest to lowest): node-level miscArgs → cluster-level miscArgs annotation → node-level spec.runtimeOptions → cluster-level spec.runtimeOptions.
Select nodes for migration: The Operator selects both storageless and storage nodes for migration in parallel:
- Storageless nodes: Multiple storageless nodes (default: 2 nodes at a time) are selected and migrated in parallel batches.
- Storage nodes: One storage node is selected at a time based on the highest used space. Selection logic varies based on the number of already migrated StoreV2 nodes:
  - First 3 storage nodes: Pool drain is performed without specifying target nodes, data can evacuate to any available node (PX-StoreV1 or PX-StoreV2) with sufficient capacity.
  - After migrating 3 nodes: A node is selected only if the PX-StoreV2 nodes collectively have sufficient free space. Pool drain is directed specifically to the PX-StoreV2 target nodes to ensure data migrates to StoreV2 storage.
Cordon the nodes: Prevent scheduling by cordoning the selected source nodes.
Drain volume attachments: Drain all volume attachments from the source nodes.
Evacuate data: Run pool drain to migrate data from storage nodes to other nodes in the cluster. For storageless nodes, a NodeDrain job is created only if there are volume attachments to drain.
Wipe and restart: Stop Portworx on the source nodes, wipe them, decommission them, and restart them with PX-StoreV2. For cloud storage, new drives are provisioned. For local storage, the same drives are reused.
Repeat: Repeat steps 3 through 7 until all nodes are converted to PX-StoreV2. Storage and storageless node migrations can run concurrently. The Operator waits only when both a storage node and storageless nodes are already migrating.

The Operator converts one storage node at a time to maintain cluster availability and resilience. During migration, the cluster operates in mixed mode with both PX-StoreV1 and PX-StoreV2 nodes. The Operator temporarily disables Autopilot to prevent interference with pool drain or rebalance operations and re-enables Autopilot after the migration completes.

Limitations

External KVDB: Migration is not supported for clusters using external KVDB.
Migration concurrency: Only one storage node migrates at a time. However, multiple storageless nodes (default is 2) can be migrated in parallel.
Three-node clusters with internal KVDB: This configuration is not supported due to quorum risk. The Operator blocks migration for clusters with fewer than 4 nodes when using internal KVDB.
- If you have a 3-node cluster with internal KVDB, you must add at least one additional node before starting the migration.
- If your cluster has more than three nodes but you have the px/metadata-node=true label on nodes that allow only three nodes to run the internal KVDB, you must update the labels to run KVDB on at least four nodes before starting the migration. For more information, see Control internal KVDB node placement.
Pool drain duration depends on the replica resynchronization speed: During migration, pool drain evacuates data from the source node and waits for replicas to resynchronize fully before moving volumes. If resynchronization is slow, for example, because of high application I/O or limited network bandwidth, pool drain can take much longer, which increases the overall migration time.
Portworx upgrades: Portworx upgrade is not supported during migration. If a Portworx upgrade is triggered while migration is InProgress, the Operator blocks the upgrade and marks the Portworx upgrade condition as Paused in the StorageCluster until migration completes.
Kubernetes upgrades: Kubernetes upgrade is not supported during migration. Perform upgrades only before starting the migration or after it completes.
Mixed mode: Supported only during migration, not for long-term deployment.
Volumes with StorageClass node constraints: Volumes created from a StorageClass with the parameters.nodes parameter set cannot be migrated automatically. Pool drain cannot relocate these volumes because the node constraint prevents placement on any other node.
Volumes with restrictive Volume Placement Strategies (VPS): If a volume has a VPS that restricts placement to specific nodes, pool drain might be unable to evacuate that volume to another node. In this case, you must relax the VPS to allow more target nodes before migration can proceed. For more information, see Volume Placement Strategies.

Supported platform and distributions

The following platforms and distributions support PX-StoreV1 to PX-StoreV2 migration:

Platform	Distribution
FlashArray	Vanilla Kubernetes, OpenShift Container Platform (OCP)
vSphere	Vanilla Kubernetes, OCP
DAS/SAN	Vanilla Kubernetes, OCP

note

For more information about supported backends with PX-StoreV2, see Supported Platforms and Distributions.

Prerequisites

Before starting the migration, make sure the following requirements are met:

Portworx version: 3.6.1 or later.
Operator version: 26.2.0 or later.
Cluster health: All nodes must be healthy and online.
Storage pools: No pools are in a degraded or offline state.
KVDB health (for internal KVDB clusters): All KVDB members must be healthy and reachable. The Operator verifies KVDB quorum before starting migration.
Cluster size: Migration is supported only on clusters with a minimum 4 nodes using internal KVDB.
Capacity: Sufficient free space to evacuate the largest node. After the migration, PX-StoreV2 reserves internal space on each pool when the pool is created for metadata and recovery purposes. The overhead varies by pool capacity and can range from approximately 6 GiB for pools smaller than 1 TiB to up to approximately 45 GiB for pools that are 8 TiB or larger. Plan for this reduction in available capacity across all pools before you start the migration.

When planning capacity, also account for the largest individual volumes. Pool drain relocates one replica to a single target node, regardless of the volume's replication factor. The target node must have free space at least equal to the volume size. If no target node can accommodate a volume, pool drain stops and migration cannot progress until you add capacity.
System metadata device: The StorageCluster is configured with a system metadata device (systemMetadataDevice) sized appropriately (typically at least 64 GiB). Note that the Operator configures cloudStorage.systemMetadataDeviceSpec automatically if you are using cloud storage.
Kernel support: All nodes must use kernel modules that PX-StoreV2 supports. For more information, see Supported kernels.
Resources: Each node must meet minimum CPU and memory requirements. For more information, see System requirements.

Run pre-migration validation

Before you start the migration, run the pre-migration validation script to verify that your cluster meets all requirements. This script performs comprehensive checks, including pod health, license validation, cluster capacity, pool health, node resources, disk capacity, and custom labels.

Prerequisites:

Python 3.8+
kubectl configured with access to your Kubernetes cluster
Access to the Portworx namespace

Download the validation script and make it executable:

curl -O https://docs.portworx.com/portworx-enterprise/scripts/migration_validator.py
chmod +x migration_validator.py

Install the required Python package:
```
pip3 install pyyaml
```
Run the validation script:
```
./migration_validator.py -n <portworx>
```
Command-line options:
- -n, --namespace: Portworx namespace (required, or the script prompts you interactively)
- -o, --output: Save the detailed report to a file (for example, -o report.txt)
- -v, --verbose: Enable debug logging to troubleshoot issues
Exit codes:
- 0: All validations passed. Ready for migration.
- 1: Critical or error issues found. Migration is blocked.
- 2: Warnings are present. Proceed with caution.
The script generates a detailed validation report that includes information such as Pod health, cluster capacity, cloud storage validation, pool health, and a migration readiness summary at the end.
Review the validation report and resolve any blocking issues before you proceed with migration.

For detailed configuration options and example output, see the validation script readme.

Migrate your cluster

After running the pre-migration validation script and fixing any blocking issues, perform the following steps to migrate your Portworx cluster from PX-StoreV1 to PX-StoreV2:

If you are using local storage, verify that your StorageCluster is configured with a system metadata device. For configuration details, see the StorageCluster CRD reference (spec.storage.systemMetadataDevice).
(Optional) Exclude specific nodes from migration by labeling each node:
```
kubectl label node <node-name> portworx.io/skip-pxStorev2-migration="true"
```
Nodes with this label are skipped during the migration process and continue running PX-StoreV1. The migration still completes for the remaining nodes. If all nodes have this label, the migration status is Skipped.
(Optional) Configure the number of storageless nodes to migrate in parallel. By default, the Operator migrates 2 storageless nodes at a time. To change this, add the following annotation to your StorageCluster:
```
metadata:
  annotations:
    portworx.io/max-storageless-nodes-to-migrate: "3"
```
Replace "3" with your desired number. This annotation affects only storageless nodes. Storage nodes are always migrated one at a time.
Edit the StorageCluster to add the portworx.io/migrate-v1-to-v2 annotation with a timestamp value:
```
apiVersion: core.libopenstorage.org/v1
kind: StorageCluster
metadata:
  name: <px-cluster>
  namespace: <portworx>
  annotations:
    portworx.io/migrate-v1-to-v2: "2 Jan 2026 3:04 PM"
...
```
The following timestamp formats are also supported:
- "2 January 2026 3:04 PM" (full month name)
- "Mon Feb 2 23:48:51 IST 2026" (terminal format)
- "02-01-2026 15:04:05" (dd-MM-yyyy HH:mm:ss)
- "2026-01-02 15:04:05" (yyyy-MM-dd HH:mm:ss)
The timestamp value determines when the migration starts:
- If the timestamp is in the future, the migration waits until that time.
- If the timestamp is in the past or current, the migration starts immediately.
note
During migration, the Operator updates the StorageCluster fields. This triggers a rolling update of Portworx pods.
Monitor the migration status:
```
kubectl get storagecluster <px-cluster> -n <portworx> -o yaml
```
```
status:
conditions:
- type: PXStoreMigration
  status: InProgress
  message: "PX store migration is in progress"
```
Check the migration condition in the status section. The migration can be in one of the following states:
- Pending: Preflight checks are running.
- Ready: Preflight passed, and the migration is ready to start.
- InProgress: Migration is actively converting nodes. Storageless nodes migrate in parallel batches, and storage nodes migrate one at a time. Both types can be migrated concurrently.
- Failed: Migration failed due to an error or was canceled. Check the message field for details.
- Completed: All nodes successfully converted to PX-StoreV2, or migration completed with some nodes skipped due to skip labels.
- Skipped: All nodes are already running PX-StoreV2, or all nodes have the skip migration label.
Monitor node conversion progress:
```
kubectl get pods -n <portworx> -l name=portworx
```
You can also check NodeDrain custom resources (CRs) to see which nodes are being migrated:
```
kubectl get nodedrain -n <portworx>
```
The output shows the migration job status for each node, including whether it's a storageless or storage node.
Check for migration events:
```
kubectl get events -n <portworx> --sort-by='.lastTimestamp'
```
Common events include:
- Preflight check results
- Node selection and migration start
- Warnings about insufficient capacity or missing nodes
- Migration progress updates

Post-migration verification

After the migration completes:

Verify all nodes are running PX-StoreV2:
```
pxctl status
```
Check for PX-StoreV2 in the output.
Check cluster health:
```
pxctl cluster provision-status
```
Verify all volumes are healthy:
```
pxctl volume list
```

Check whether any nodes were skipped:

kubectl get nodes -l portworx.io/skip-pxStorev2-migration=true

If nodes were skipped, they are still running PX-StoreV1. To migrate them, remove the skip label and re-annotate the StorageCluster to restart migration:

kubectl label node <node-name> portworx.io/skip-pxStorev2-migration-
kubectl annotate storagecluster <px-cluster> -n <portworx> \
  portworx.io/migrate-v1-to-v2="$(date '+%d %b %Y %I:%M %p')" --overwrite

Remove the migration annotation (optional):

kubectl annotate storagecluster <px-cluster> -n <portworx> \
  portworx.io/migrate-v1-to-v2-

The Operator automatically removes the following from your StorageCluster after all nodes complete migration:
- The allow_mixed_mode=1 runtime option
- Node labels portworx.io/pxStorev2-migrated-node
The -T PX-StoreV2 flag is retained in the StorageCluster CRD.

note
If some nodes were skipped due to skip labels, the mixed mode configuration remains in place until you complete migration for those nodes or manually remove the configuration.

Troubleshooting

Pool drain stops because no target node has enough free space to accommodate a large volume. The source node is cordoned and volume attachments are disabled, but migration doesn't progress.

Symptom: Pool drain logs contain free size must be >= X.

Recovery:
1. Get the pool drain job ID from the NodeDrain CR:
```
kubectl get nodedrain -n <portworx> -o wide
```
2. Cancel the pool drain job. After cancellation, the migration marks itself as Failed, the Operator uncordons the source node, and volume attachments are re-enabled. Wait a few minutes for any affected pods to return to the Running state.
```
pxctl service pool drain cancel --job-id <job-id>
```
3. Add capacity so that at least one target node can accommodate the volume:
  - Cloud drives: Expand the pool on a target node:
    pxctl service pool expand -o <mode> -s <size> -u <pool-uid>
  - Preprovisioned drives: Either add a new node with sufficient capacity to the cluster, or decommission an existing node (either an already-migrated PX-StoreV2 node or a PX-StoreV1 node), add drives to it by updating the StorageCluster node spec, and then recommission it. Move volume replicas off the node before decommissioning. A recommissioned PX-StoreV1 node restarts as PX-StoreV2.
4. Restart the migration:
```
kubectl annotate storagecluster <px-cluster> -n <portworx> \
  portworx.io/migrate-v1-to-v2="$(date '+%d %b %Y %I:%M %p')" --overwrite
```
If the migration fails, follow these steps to troubleshoot:
1. Check the StorageCluster status for error details:
```
kubectl get storagecluster <px-cluster> -n <portworx> -o jsonpath='{.status.conditions[?(@.type=="PXStoreMigration")]}'
```
2. Review the Operator logs:
```
kubectl logs -n <portworx> -l name=portworx-operator
```
3. Check the status of NodeDrain CRs:
```
kubectl get nodedrain -n <portworx> -o yaml
kubectl describe nodedrain <node-drain> -n <portworx>
```
  The NodeDrain status determines the recovery path:
  - Canceled: The migration automatically marks itself as Failed and sets LastMigrationState to enable restart. The Operator automatically uncordons the affected nodes. When you restart the migration, canceled NodeDrain custom resources (CRs) are reset to NotStarted.
  - Failed: Review the NodeDrain CR details to determine the recovery steps before restarting:
    - Node started in a storageless state: Review the Portworx logs on the affected node and the NodeDrain CR for error details. If you cannot resolve the issue, contact support. After you restart the migration, the Operator assumes that the node is in the correct state, skips it, and continues with the next node. To retry migration on the same node instead, delete the NodeDrain CR and remove the portworx.io/pxStorev2-migrated-node label from the node before restarting:
      kubectl delete nodedrain <node-drain-name> -n <portworx> kubectl label node <node-name> portworx.io/pxStorev2-migrated-node-
    - Node started as PX-StoreV1 (btrfs) again: The node was not successfully converted. Delete the failed NodeDrain custom resource (CR) and remove the portworx.io/pxStorev2-migrated-node label from the affected node before you restart the migration:
      kubectl delete nodedrain <node-drain-name> -n <portworx> kubectl label node <node-name> portworx.io/pxStorev2-migrated-node-
4. Restart the migration by updating the annotation timestamp:
```
kubectl annotate storagecluster <px-cluster> -n <portworx> \
  portworx.io/migrate-v1-to-v2="$(date '+%d %b %Y %I:%M %p')" --overwrite
```
When you restart a failed migration, the Operator resumes from the migration phase recorded at the time of failure: either Pending to re-run preflight checks, or InProgress to continue node migration. Nodes already running PX-StoreV2 are not re-migrated. Cancelled NodeDrain CRs are reset to NotStarted and retried; failed NodeDrain CRs are marked Completed and skipped so migration continues with the next node.

Known issues

On OpenShift Container Platform (OCP) environments, NFS kernel threads can get stuck in an uninterruptible sleep state. These stuck threads prevent unmounting of encrypted SharedV4 volumes even when there is no active I/O on the device. As a result, the node drain-attachments job times out and migration stalls on the affected node.

Symptoms:
- The drain-attachments job shows FAILED with the message job timed out and node still has volume attachments.
- umount on the affected volume returns target is busy even though the inflight I/O count is zero.
- NFS kernel threads (mount.nfs) appear in the D (uninterruptible sleep) state in ps output.
Workaround: Reboot the affected node. This releases the stuck kernel NFS threads, allowing migration to proceed.
On OCP with pre-provisioned drives, the migration preflight DaemonSet creates pods that use the px-pre-flight service account with hostPID: true. The Portworx Security Context Constraint (SCC) is configured with allowHostPID: false at installation time and does not grant the required privileges. The OCP SCC admission controller blocks pod creation, which causes the preflight stage to time out and the migration to fail.

Workaround: Before starting the migration, grant the privileged SCC to the px-pre-flight service account:
```
oc adm policy add-scc-to-user privileged -z px-pre-flight -n <portworx>
```

How the migration works​

Limitations​

Supported platform and distributions​

Prerequisites​

Run pre-migration validation​

Migrate your cluster​

Post-migration verification​

Troubleshooting​

Known issues​