Migration with Stork on Kubernetes
This document will walk you through how to migrate your Portworx volumes between clusters with Stork on Kubernetes.
Prerequisites
Before we begin, please make sure the following prerequisites are met:
- Version: To take advantage of the features described in this document, the source AND destination clusters must have Stork 2.12.0 or later, Operator 1.10.0 or later, and Portworx Enterprise 2.11.0 or later.
- Stork helper :
storkctl
is a command-line tool for interacting with a set of scheduler extensions. - Secret Store : Make sure you have configured a secret store on both clusters. This will be used to store the credentials for the objectstore.
- Network Connectivity:
- Portworx: Port 9001 on the destination cluster should be reachable by the source cluster. For OpenShift, Port 17001 should be reachable.
- Kubernetes API End Point: Port 6443, 443, or 8443 should be reachable. Check with your Kubernetes administrator to determine which port you need.
Parts of this document require you to specify a <namespace>
. Specify the namespace where Portworx is installed. Typically, Portworx is installed in either the kube-system
or portworx
namespace.
Download storkctl
Always use the latest storkctl
binary tool by downloading it from the current running Stork container.
Perform the following steps to download storkctl
from the Stork pod:
Linux:
STORK_POD=$(kubectl get pods -n <namespace> -l name=stork -o jsonpath='{.items[0].metadata.name}') &&
kubectl cp -n <px-namespace> $STORK_POD:/storkctl/linux/storkctl ./storkctl
sudo mv storkctl /usr/local/bin &&
sudo chmod +x /usr/local/bin/storkctlOS X:
STORK_POD=$(kubectl get pods -n <namespace> -l name=stork -o jsonpath='{.items[0].metadata.name}') &&
kubectl cp -n <px-namespace> $STORK_POD:/storkctl/darwin/storkctl ./storkctl
sudo mv storkctl /usr/local/bin &&
sudo chmod +x /usr/local/bin/storkctlWindows:
Copy
storkctl.exe
from the stork pod:STORK_POD=$(kubectl get pods -n <px-namespace> -l name=stork -o jsonpath='{.items[0].metadata.name}') &&
kubectl cp -n <px-namespace> $STORK_POD:/storkctl/windows/storkctl.exe ./storkctl.exeMove
storkctl.exe
to a directory in your PATH.
In Kubernetes, you must define a trust object called ClusterPair. Portworx requires this object to communicate with the destination cluster. The ClusterPair object pairs the Portworx storage driver with the Kubernetes scheduler, allowing volumes and resources to be migrated between clusters.
The ClusterPair is generated and used in the following way:
- It requires
storkctl
to be running on the Kubernetes control plane node in both the destination and source clusters - The ClusterPair spec is generated on the destination cluster
- The generated spec is then applied on the source cluster
Perform the following steps to create a cluster pair:
You must run the pxctl
commands in this document either on your worker nodes directly, or from inside the Portworx containers on your Kubernetes control plane nodes.
Create object store credentials for cloud clusters
Create object store credentials on your source and destination clusters. The options you use to create your object store credentials differ based on which object store you use.
You must create object store credentials on both the destination and source clusters before you can create a cluster pair.
Create Amazon s3 or s3 compatible object store credentials
Find the UUID of your destination cluster by running the following command:
PX_POD=$(kubectl get pods -l name=portworx -n <namespace> -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n <namespace> -- /opt/pwx/bin/pxctl status | grep UUID | awk '{print $3}'Enter the
pxctl credentials create
command, specifying the following:- The
--provider
flag with the name of the cloud provider (s3
). - The
--s3-access-key
flag with your secret access key - The
--s3-secret-key
flag with your access key ID - The
--s3-region
flag with the name of the S3 region (us-east-1
) - The
--s3-endpoint
flag with the name of the endpoint (s3.amazonaws.com
) - The optional
--s3-storage-class
flag with either theSTANDARD
orSTANDARD-IA
value, depending on which storage class you prefer clusterPair_<UUID_of_destination_cluster>
Enter the UUID of your destination cluster collected in step 1.
/opt/pwx/bin/pxctl credentials create \
--provider s3 \
--s3-access-key <aws_access_key> \
--s3-secret-key <aws_secret_key> \
--s3-region us-east-1 \
--s3-endpoint s3.amazonaws.com \
--s3-storage-class STANDARD \
clusterPair_<UUID_of_destination_cluster>- The
Create Microsoft Azure credentials
Find the UUID of your destination cluster by running the following command:
PX_POD=$(kubectl get pods -l name=portworx -n <namespace> -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n <namespace> -- /opt/pwx/bin/pxctl status | grep UUID | awk '{print $3}'Enter the
pxctl credentials create
command, specifying the following:--provider
asazure
--azure-account-name
with the name of your Azure account--azure-account-key
with your Azure account keyclusterPair_<UUID_of_destination_cluster>
Enter the UUID of your destination cluster collected in step 1.
/opt/pwx/bin/pxctl credentials create \
--provider azure \
--azure-account-name <your_azure_account_name> \
--azure-account-key <your_azure_account_key> \
clusterPair_<UUID_of_destination_cluster>
Create Google Cloud Platform credentials
Find the UUID of your destination cluster by running the following command:
PX_POD=$(kubectl get pods -l name=portworx -n <namespace> -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n <namespace> -- /opt/pwx/bin/pxctl status | grep UUID | awk '{print $3}'Enter the
pxctl credentials create
command, specifying the following:--provider
asgoogle
--google-project-id
with the string of your Google project ID--google-json-key-file
with the filename of your GCP JSON key fileclusterPair_<UUID_of_destination_cluster>
Enter the UUID of your destination cluster collected in step 1.
/opt/pwx/bin/pxctl credentials create \
--provider google \
--google-project-id <your_google_project_ID> \
--google-json-key-file <your_GCP_JSON_key_file> \
clusterPair_<UUID_of_destination_cluster>
Using Rancher Projects with ClusterPair
If you are not using Rancher, skip to the next section.
Rancher has a concept of Projects that allow grouping of resources and Kubernetes namespaces. Depending on the resource and how it is created, Rancher adds the following label or annotation:
field.cattle.io/projectID: <project-short-UUID>
The projectID
uniquely identifies the project, and the annotation or label on the Kubernetes object provides a way to tie a Kubernetes object back to a Rancher project.
From version 2.11.2 or newer, Stork has the capability to map projects from the source cluster to the destination cluster when it migrates Kubernetes resources. It will ensure that the following are transformed when migrating Kubernetes resources to a destination cluster:
- Labels and annotations for projectID
field.cattle.io/projectID
on any Kubernetes resource on the source cluster are transformed to their respective projectIDs on the destination cluster. - Namespace Selectors on a NetworkPolicy object which refer to the
field.cattle.io/projectID
label will be transformed to their respective projectIDs on the destination cluster. - Namespace Selectors on a Pod object (Kubernetes version 1.24 or newer) which refer to the
field.cattle.io/projectID
label will be transformed to their respective projectIDs on the destination cluster.
- Rancher project mappings are supported only with Stork version 2.11.2 or newer.
- All the Rancher projects need to be created on both the source and the destination cluster.
While creating the ClusterPair, use the argument --project-mappings
to indicate which projectID on the source cluster maps to a projectID on the destination cluster.
For example:
storkctl generate clusterpair -n <migrationnamespace> <remotecluster> --project-mappings <projectID-A1>=<projectID-A2>,<projectID-B1>=<projectID-B2>
The project mappings are provided as a comma-separate key=value
pairs. In this example, projectID-A1
on source cluster maps to projectID-A2
on the destination cluster, while projectID-B1
on the source cluster maps to projectID-B2
on the destination cluster.
Generate a ClusterPair from the destination cluster
The <migrationnamespace>
in this example is an admin namespace. The <migrationnamespace>
can also be a non-admin namespace. By default, kube-system
is the default admin namespace, but another namespace can also be defined as an admin namespace. An admin namespace is a special namespace because any user with access to that namespace will be able to migrate any other namespace. If you only have access to a non-admin namespace, you can migrate only that namespace. For more information on setting up an admin namespace, see cluster-admin-namespace.
To generate the ClusterPair spec, run the following command on the destination cluster:
storkctl generate clusterpair -n <migrationnamespace> <remotecluster>
Here, remotecluster
is the Kubernetes object that will be created on the source cluster representing the pair relationship, and migrationnamespace
is the Kubernetes namespace of the source cluster that you want to migrate to the destination cluster.
During the actual migration, you will reference this name to identify the destination of your migration.
If you are using PX-Security on both the source and destination clusters, you will need to add the following two annotations to the ClusterPair
and MigrationSchedule
:
annotations:
openstorage.io/auth-secret-namespace -> Points to the namespace where the k8s secret holding the auth token resides.
openstorage.io/auth-secret-name -> Points to the name of the k8s secret which holds the auth token.
Save the resulting spec to a file named clusterpair.yaml
.
Example:
storkctl generate clusterpair -n <migrationnamespace> <remotecluster> -o yaml > clusterpair.yaml
apiVersion: stork.libopenstorage.org/v1alpha1
kind: ClusterPair
metadata:
creationTimestamp: null
name: <remotecluster>
namespace: <migrationnamespace>
annotations:
# Add the below annotations when PX-Security is enabled on both the clusters
#openstorage.io/auth-secret-namespace -> Points to the namespace where the k8s secret holding the auth token resides
#openstorage.io/auth-secret-name -> Points to the name of the k8s secret which holds the auth token.
spec:
config:
clusters:
kubernetes:
LocationOfOrigin: /etc/kubernetes/admin.conf
certificate-authority-data: <CA_DATA>
server: https://192.168.56.74:6443
contexts:
kubernetes-admin@kubernetes:
LocationOfOrigin: /etc/kubernetes/admin.conf
cluster: kubernetes
user: kubernetes-admin
current-context: kubernetes-admin@kubernetes
preferences: {}
users:
kubernetes-admin:
LocationOfOrigin: /etc/kubernetes/admin.conf
client-certificate-data: <CLIENT_CERT_DATA>
client-key-data: <CLIENT_KEY_DATA>
options:
<insert_storage_options_here>: ""
status:
remoteStorageId: ""
schedulerStatus: ""
storageStatus: ""
Get the destination cluster token
On the destination cluster, run the following command from one of the Portworx nodes to get the cluster token. You'll need this token in later steps:
PX_POD=$(kubectl get pods -l name=portworx -n <namespace> -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n <namespace> -- /opt/pwx/bin/pxctl cluster token show
Insert Storage Options
To pair the storage, specify the following fields in the options
section of the clusterpair.yaml
file saved previously:
ip
: The IP address of any of the remote Portworx nodes. If using external load balancer, specify the IP address of the external load balancer.port
: The port of the remote Portworx node. See Network Connectivity in the Prerequisites section of this document.token
: The token of the destination cluster, which you obtained in the previous step.
See example below:
options:
token: "34b2fd8df839650ed8f9b5cd5a70e1ca6d79c1ebadb5f50e29759525a20aee7daa5aae35e1e23729a4d9f673ce465dbe3679032d1be7a1bcc489049a3c23a460"
ip: "10.13.21.125"
port: "9001"
Apply the ClusterPair spec on the source cluster
To create the ClusterPair, apply the ClusterPair YAML spec on the source cluster. Run the following command from a location where you have kubectl
access to the source cluster:
kubectl create -f clusterpair.yaml
Verify the ClusterPair
To verify that you have generated the ClusterPair and that it is ready, run the following command:
storkctl get clusterpair -n <migrationnamespace>
NAME STORAGE-STATUS SCHEDULER-STATUS CREATED
remotecluster Ready Ready 09 Nov 22 00:22 UTC
On a successful pairing, you should see the STORAGE-STATUS
and SCHEDULER-STATUS
as Ready
.
If you see an error instead, you can get more information by running the following command:
kubectl describe clusterpair remotecluster -n <migrationnamespace>
Migrating volumes and resources
Once the pairing is configured, applications can be migrated repeatedly to the destination cluster.
If your cluster has a DR license applied to it, you can perform migrations only in DR mode; this includes operations involving the pxctl cluster migrate
command.
Start a migration
- Create a file called
migration.yaml
file, specifying the following fields and values:
apiVersion: as
stork.libopenstorage.org/v1alpha1
kind: as
Migration
metadata.name: with the name of the object that performs the migration
metadata.namespace: with the name of the namespace in which you want to create the object
spec.clusterPair: with the name of the
ClusterPair
object created in the Generate a ClusterPair from the destination cluster sectionspec.includeResources: with a boolean value specifying if the migration should include PVCs and other applications specs. If you set this field to
false
, Portworx will only migrate your volumes.spec.startApplications: with a boolean value specifying if Portworx should automatically start applications on the destination cluster. If you set this field to
false
, then theDeployment
andStatefulSet
objects on the destination cluster will be set to zero. Note that, on the destination cluster, Portworx uses thestork.openstorage.org/migrationReplicas
annotation to store the number of replicas from the source cluster.spec.namespaces: with the list of namespaces you want to migrate
spec.adminClusterPair: with the name of the
ClusterPair
object created in the admin namespace that Portworx should use to migrate your cluster-scoped resources. Use this if your regular users lack permission to create or delete cluster-scoped resources on the destination cluster. If you don't specify this field, then Stork will use the object specified in thespec.clusterPair
field. This feature was introduced in Stork version 2.3.0.spec.purgeDeletedResources: with a boolean value specifying if Stork should automatically purge a resource from the destination cluster when you delete it from the source cluster. The default value is
false
. This feature was introduced in Stork version 2.3.2.apiVersion: stork.libopenstorage.org/v1alpha1
kind: Migration
metadata:
name: <YOUR_MIGRATION_OBJECT>
namespace: <YOUR_MIGRATION_NAMESPACE>
spec:
clusterPair: <YOUR_CLUSTER_PAIR>
includeResources: true
startApplications: true
namespaces:
- <NAMESPACE_TO_MIGRATE>
adminClusterPair: <YOUR_ADMIN_CLUSTER_PAIR>
purgeDeletedResources: false
Apply the spec by entering the following command:
kubectl apply -f migration.yaml
Stork users: You can run the following example command to create a migration:
storkctl create migration <YOUR_MIGRATION_OBJECT> --clusterPair <YOUR_CLUSTER_PAIR> --namespaces <NAMESPACE_TO_MIGRATE> --includeResources --startApplications -n <YOUR_NAMESPACE>
Migration scope
Currently, you can only migrate namespaces in which the object is created. You can also designate one namespace as an admin namespace. This will allow an admin who has access to that namespace to migrate any namespace from the source cluster. Instructions for setting this admin namespace to stork
can be found in Set up a cluster admin namespace.
Monitoring a migration
Once the migration has been started using the previous commands, you can check the status using storkctl
. For example:
storkctl get migration -n <migrationnamespace>
First, you should see something like the following:
NAME CLUSTERPAIR STAGE STATUS VOLUMES RESOURCES CREATED
zoomigrationschedule-interval-2022-12-12-200210 remotecluster Volumes InProgress 0/3 0/0 12 Dec 22 11:45 PST
If the migration is successful, the STAGE
will go from Volumes
to Application
to Final
.
Here is how the example output of a successful migration looks:
NAME CLUSTERPAIR STAGE STATUS VOLUMES RESOURCES CREATED ELAPSED
zoomigrationschedule-interval-2022-12-12-200210 remotecluster Final Successful 3/3 10/10 12 Dec 22 12:02 PST 1m23s
Troubleshooting
If there is a failure or you want more information about what resources were migrated you can describe
the migration object using kubectl
:
kubectl describe migration <migrationschedulename> -n <migrationnamespace>
Name: zoomigrationschedule-interval-2022-12-12-220239
Namespace: zookeeper
Labels: <none>
Annotations: stork.libopenstorage.org/migration-schedule-name: zoomigrationschedule
API Version: stork.libopenstorage.org/v1alpha1
Kind: Migration
Metadata:
Creation Timestamp: 2022-12-12T22:02:39Z
Finalizers:
stork.libopenstorage.org/finalizer-cleanup
Generation: 202
Managed Fields:
API Version: stork.libopenstorage.org/v1alpha1
Fields Type: FieldsV1
fieldsV1:
......
...........
Manager: stork
Operation: Update
Time: 2022-12-12T22:02:40Z
Owner References:
API Version: stork.libopenstorage.org/v1alpha1
Kind: MigrationSchedule
Name: zoomigrationschedule
UID: 5ae0cdb9-e4ba-4562-8094-8d7763d99ed1
Resource Version: 31906052
UID: 89c0ae5e-f08a-454f-b7b3-59951662ab0b
Spec:
Admin Cluster Pair:
Cluster Pair: remotecluster
Include Network Policy With CIDR: false
Include Optional Resource Types: <nil>
Include Resources: true
Include Volumes: true
Namespaces:
zookeeper
Post Exec Rule:
Pre Exec Rule:
Purge Deleted Resources: false
Selectors: <nil>
Skip Deleted Namespaces: <nil>
Skip Service Update: false
Start Applications: false
Status:
Finish Timestamp: 2022-12-12T22:04:40Z
Resource Migration Finish Timestamp: 2022-12-12T22:04:40Z
Resources:
Group: core
Kind: PersistentVolume
Name: pvc-0a330e8b-d5f3-4c65-bca6-3eb37adcfb2d
Namespace:
Reason: Resource migrated successfully
Status: Successful
Version: v1
Group: core
Kind: PersistentVolume
Name: pvc-62ad252d-2218-4d33-9c57-8e61641b1a8c
Namespace:
Reason: Resource migrated successfully
Status: Successful
Version: v1
Group: core
Kind: PersistentVolume
Name: pvc-fdd559fa-b42c-4424-9699-cf13bc14a77e
Namespace:
Reason: Resource migrated successfully
Status: Successful
Version: v1
Group: core
Kind: PersistentVolumeClaim
Name: datadir-zk-0
Namespace: zookeeper
Reason: Resource migrated successfully
Status: Successful
Version: v1
Group: core
Kind: PersistentVolumeClaim
Name: datadir-zk-1
Namespace: zookeeper
Reason: Resource migrated successfully
Status: Successful
Version: v1
Group: core
Kind: PersistentVolumeClaim
Name: datadir-zk-2
Namespace: zookeeper
Reason: Resource migrated successfully
Status: Successful
Version: v1
Group: core
Kind: Service
Name: zk-cs
Namespace: zookeeper
Reason: Resource migrated successfully
Status: Successful
Version: v1
Group: core
Kind: Service
Name: zk-hs
Namespace: zookeeper
Reason: Resource migrated successfully
Status: Successful
Version: v1
Group: apps
Kind: StatefulSet
Name: zk
Namespace: zookeeper
Reason: Resource migrated successfully
Status: Successful
Version: v1
Group: policy
Kind: PodDisruptionBudget
Name: zk-pdb
Namespace: zookeeper
Reason: Resource migrated successfully
Status: Successful
Version: v1
Stage: Final
Status: Successful
Summary:
Elapsed Time For Resource Migration: 7s
Elapsed Time For Volume Migration: 1m54s
Num Of Migrated Resources: 10
Num Of Migrated Volumes: 3
Total Bytes Migrated: 12288
Total Number Of Resources: 10
Total Number Of Volumes: 3
Volume Migration Finish Timestamp: 2022-12-12T22:04:33Z
Volumes:
Bytes Total: 4096
Namespace: zookeeper
Persistent Volume Claim: datadir-zk-0
Reason: Migration successful for volume
Status: Successful
Volume: pvc-0a330e8b-d5f3-4c65-bca6-3eb37adcfb2d
Bytes Total: 4096
Namespace: zookeeper
Persistent Volume Claim: datadir-zk-1
Reason: Migration successful for volume
Status: Successful
Volume: pvc-62ad252d-2218-4d33-9c57-8e61641b1a8c
Bytes Total: 4096
Namespace: zookeeper
Persistent Volume Claim: datadir-zk-2
Reason: Migration successful for volume
Status: Successful
Volume: pvc-fdd559fa-b42c-4424-9699-cf13bc14a77e
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Failed 20m stork Error migrating volumes: Operation cannot be fulfilled on migrations.stork.libopenstorage.org "zoomigrationschedule-interval-2022-12-12-220239": the object has been modified; please apply your changes to the latest version and try again
Normal Successful 19m (x13 over 19m) stork Volume pvc-0a330e8b-d5f3-4c65-bca6-3eb37adcfb2d migrated successfully
Normal Successful 19m (x11 over 19m) stork Volume pvc-62ad252d-2218-4d33-9c57-8e61641b1a8c migrated successfully
Pre and Post Exec rules
Similar to snapshots, a PreExec and PostExec rule can be specified when creating a Migration object. This will result in the PreExec rule being run before the migration is triggered and the PostExec rule to be run after the Migration has been triggered. If the rules do not exist, the Migration will log an event and will stop.
If the PreExec rule fails for any reason, it will log an event against the object and retry. The Migration will not be marked as failed.
If the PostExec rule fails for any reason, it will log an event and mark the Migration as failed. It will also try to cancel the migration that was started from the underlying storage driver.
In the example below with mysql
, to add pre and post rules to our migration, we could edit our migration.yaml
file like this:
apiVersion: stork.libopenstorage.org/v1alpha1
kind: Migration
metadata:
name: mysqlmigration
namespace: mysql
spec:
clusterPair: remotecluster
includeResources: true
startApplications: true
preExecRule: mysql-pre-rule
postExecRule: mysql-post-rule
namespaces:
- mysql