Kubemotion with stork on Kubernetes


This document will walk you through how to migrate your PX volumes between clusters with Stork on Kubernetes.

Prerequisites

Before we begin, please make sure the following prerequisites are met:

  • Version: The source AND destination clusters need PX-Enterprise v2.0 or later release. As future releases are made, the two clusters can have different PX-Enterprise versions (e.g. v2.1 and v2.3).

  • Stork v2.0+ is required on the source cluster.

  • Stork helper : storkctl is a command-line tool for interacting with a set of scheduler extensions.

    The following steps can be used to download storkctl from the stork pod:

    • Linux:

      STORK_POD=$(kubectl get pods -n kube-system -l name=stork -o jsonpath='{.items[0].metadata.name}') &&
      kubectl cp -n kube-system $STORK_POD:/storkctl/linux/storkctl ./storkctl
      sudo mv storkctl /usr/local/bin &&
      sudo chmod +x /usr/local/bin/storkctl
    • OS X:

      STORK_POD=$(kubectl get pods -n kube-system -l name=stork -o jsonpath='{.items[0].metadata.name}') &&
      kubectl cp -n kube-system $STORK_POD:/storkctl/darwin/storkctl ./storkctl
      sudo mv storkctl /usr/local/bin &&
      sudo chmod +x /usr/local/bin/storkctl
    • Windows:

      • Copy storkctl.exe from the stork pod: text STORK_POD=$(kubectl get pods -n kube-system -l name=stork -o jsonpath='{.items[0].metadata.name}') && kubectl cp -n kube-system $STORK_POD:/storkctl/windows/storkctl.exe ./storkctl.exe
      • Move storkctl.exe to a directory in your PATH

  • Secret Store : Make sure you have configured a secret store on both clusters. This will be used to store the credentials for the objectstore.

  • Network Connectivity: Ports 9001 and 9010 on the destination cluster should be reachable by the source cluster.

Generate and Apply a ClusterPair Spec

In Kubernetes, you must define a trust object called ClusterPair. Portworx requires this object to communicate with the destination cluster. The ClusterPair object pairs the Portworx storage driver with the Kubernetes scheduler, allowing the volumes and resources to be migrated between clusters.

The ClusterPair is generated and used in the following way:

  • The ClusterPair spec is generated on the destination cluster.
  • The generated spec is then applied on the source cluster

Perform the following steps to create a cluster pair:

Create object store credentials for cloud clusters

If you are running Kubernetes on-premises, you may skip this section. If your Kubernetes clusters are on the cloud, you must create object credentials on both the destination and source clusters before you can create a cluster pair.

The options you use to create your object store credentials differ based on which object store you use:

Create Amazon s3 credentials

Create the credentials by entering the pxctl credentials create command, specifying the following:

  • --provider as s3
  • --s3-access-key with your aws access key
  • --s3-secret-key with your aws secret key
  • --s3-region with your region
  • --s3-endpoint with s3.amazonaws.com
  • clusterPair_ with the UUID of your destination cluster
/opt/pwx/bin/pxctl credentials create --provider s3 --s3-access-key <aws_access_key> --s3-secret-key <aws_secret_key> --s3-region us-east-1  --s3-endpoint s3.amazonaws.com clusterPair_<UUID_of_destination_cluster>

Create Microsoft Azure credentials

Create the credentials by entering the pxctl credentials create command, specifying the following:

  • --provider as azure
  • --azure-account-name with the name of your Azure account
  • --azure-account-key with your Azure account key
  • clusterPair_ with the UUID of your destination cluster appended
/opt/pwx/bin/pxctl credentials create --provider azure --azure-account-name <your_azure_account_name> --azure-account-key <your_azure_account_key> clusterPair_<UUID_of_destination_cluster>

Create Google Cloud Platform credentials

Create the credentials by entering the pxctl credentials create command, specifying the following:

  • --provider as google
  • --google-project-id with the string of your Google project ID
  • --google-json-key-file with the filename of your GCP JSON key file
  • clusterPair_ with the UUID of your destination cluster appended
/opt/pwx/bin/pxctl credentials create --provider google --google-project-id <your_google_project_ID> --google-json-key-file <your_GCP_JSON_key_file> clusterPair_<UUID_of_destination_cluster>

Generate a ClusterPair on the destination cluster

To generate the ClusterPair spec, run the following command on the destination cluster:

storkctl generate clusterpair -n migrationnamespace remotecluster

Here, the name (remotecluster) is the Kubernetes object that will be created on the source cluster representing the pair relationship.

During the actual migration, you will reference this name to identify the destination of your migration.

apiVersion: stork.libopenstorage.org/v1alpha1
kind: ClusterPair
metadata:
    creationTimestamp: null
    name: remotecluster
    namespace: migrationnamespace
spec:
   config:
      clusters:
         kubernetes:
            LocationOfOrigin: /etc/kubernetes/admin.conf
            certificate-authority-data: <CA_DATA>
            server: https://192.168.56.74:6443
      contexts:
         kubernetes-admin@kubernetes:
            LocationOfOrigin: /etc/kubernetes/admin.conf
            cluster: kubernetes
            user: kubernetes-admin
      current-context: kubernetes-admin@kubernetes
      preferences: {}
      users:
         kubernetes-admin:
            LocationOfOrigin: /etc/kubernetes/admin.conf
            client-certificate-data: <CLIENT_CERT_DATA>
            client-key-data: <CLIENT_KEY_DATA>
    options:
       <insert_storage_options_here>: ""
status:
  remoteStorageId: ""
  schedulerStatus: ""
  storageStatus: ""

Show your destination cluster token

On the destination cluster, run the following command from one of the Portworx nodes to get the cluster token. You’ll need this token in later steps:

pxctl cluster token show

Update ClusterPair with storage options

Next, let’s edit the ClusterPair spec. Under spec.options, add the following Portworx clusterpair information:

  1. ip: the IP address of one of the Portworx nodes on the destination cluster
  2. port: the port on which the Portworx API server is listening for requests. Default is 9001 if not specified
  3. token: the cluster token generated in the previous step

The updated ClusterPair should look like this:

apiVersion: stork.libopenstorage.org/v1alpha1
kind: ClusterPair
metadata:
  creationTimestamp: null
  name: remotecluster
  namespace: migrationnamespace
spec:
  config:
      clusters:
        kubernetes:
          LocationOfOrigin: /etc/kubernetes/admin.conf
          certificate-authority-data: <CA_DATA>
          server: https://192.168.56.74:6443
      contexts:
        kubernetes-admin@kubernetes:
          LocationOfOrigin: /etc/kubernetes/admin.conf
          cluster: kubernetes
          user: kubernetes-admin
      current-context: kubernetes-admin@kubernetes
      preferences: {}
      users:
        kubernetes-admin:
          LocationOfOrigin: /etc/kubernetes/admin.conf
          client-certificate-data: <CLIENT_CERT_DATA>
          client-key-data: <CLIENT_KEY_DATA>
  options:
      ip: "<ip-address-of-node-in-the-destination-cluster>"
      port: "<port_of_remote_px_node_default_9001>"
      token: "<token_generated_from_destination_cluster>"
status:
  remoteStorageId: ""
  schedulerStatus: ""
  storageStatus: ""

Instead of the IP address of the node in the destination cluster, you can use the hostname, or any DNS name.

In the updated spec, ensure values for all fields under options are quoted.

Copy and save this to a file called clusterpair.yaml on the source cluster.

Creating the ClusterPair

Apply the generated ClusterPair on the source cluster

On the source cluster, create the clusterpair by applying clusterpair.yaml:

kubectl apply -f clusterpair.yaml
clusterpair.stork.libopenstorage.org/remotecluster created

Note that, when the ClusterPair gets created, Portworx also creates a 100 GiB volume called ObjectstoreVolume. If you plan to migrate volumes that are significanlty larger than 100GiB, make sure you check out first the Migrating Large Volumes section.

Verifying the Pair status

Once you apply the above spec on the source cluster, you should be able to check the status of the pairing:

storkctl get clusterpair
NAME               STORAGE-STATUS   SCHEDULER-STATUS   CREATED
remotecluster      Ready            Ready              26 Oct 18 03:11 UTC

On a successful pairing, you should see the “Storage Status” and “Scheduler Status” as “Ready”:

If so, you’re all set and ready to migrate.

Troubleshooting

If instead, you see an error, you should get more information by running:

kubectl describe clusterpair remotecluster
You might need to perform additional steps for GKE and EKS

Migrating Volumes and Resources

Once the pairing is configured, applications can be migrated repeatedly to the destination cluster.

NOTE: If your cluster has a DR license applied to it, you can only perform migrations in DR mode; this includes operations involving the pxctl cluster migrate command.

Migrating Large Volumes

When the clusterpair gets created, Portworx automatically creates a 100G volume named ObjectstoreVolume. If you attempt to migrate a volume significantly larger than 100G, you will find out that the ObjectStore volume doesn’t provide sufficient disk space and the migration will fail.

As an example, say you want to migrate a 1 TB volume. If so, you would need to update the size of the ObjectstoreVolume by running:

pxctl volume update --size 1005 ObjectstoreVolume
Kubemotion uses compression when transferring data. So, if the data is easily compressible, a 100G Objectstore could allow you to transfer more than 100G of data. However, there is no way to tell beforehand how much the data will be compressed. In our example, we migrated a few docker images that were not easily compressible. Thus, reallocating the Objectstore to > 1TB was a safe thing to do.

Here’s how you can check if a migration failed due to insufficient space:

storkctl -n nexus get migration nexusmigration -o yaml
apiVersion: v1
items:
- metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"stork.libopenstorage.org/v1alpha1","kind":"Migration","metadata":{"annotations":{},"name":"nexusmigration","namespace":"nexus"},"spec":{"clusterPair":"remoteclusteroldgreatdanetonewgreatdane","includeResources":true,"namespaces":["nexus"],"startApplications":true}}
    creationTimestamp: "2019-07-02T23:49:13Z"
    generation: 1
    name: nexusmigration
    namespace: nexus
    resourceVersion: "77951964"
    selfLink: /apis/stork.libopenstorage.org/v1alpha1/namespaces/nexus/migrations/nexusmigration
    uid: fec861ca-9d23-11e9-beb8-0cc47ab5f9a2
  spec:
    clusterPair: remoteclusteroldgreatdanetonewgreatdane
    includeResources: true
    namespaces:
    - nexus
    selectors: null
    startApplications: true
  status:
    resources: null
    stage: Final
    status: Failed
    volumes:
    - namespace: nexus
      persistentVolumeClaim: nexus-pvc
      reason: "Migration Backup failed for volume: Backup failed: XMinioStorageFull:
        Storage backend has reached its minimum free disk threshold. Please delete
        a few objects to proceed.\n\tstatus code: 500, request id: 15ADBC3B90C3A97F,
        host id: "
      status: Failed
      volume: pvc-9b776615-3f5e-11e8-83b6-0cc47ab5f9a2
kind: List
metadata: {}

If you see an error similar to the one above, you should increase the size of the ObjectstoreVolume and restart the migration.

Alternatively, you can use your own cloud (S3, Azure, or Google) instead of ObjectStore on the destination cluster. Note that the credentials must be named clusterPair_<clusterUUID_of_destination> and you are required to create them on both the source and the destination cluster.

Starting a migration

Using a spec file

In order to make the process schedulable and repeatable, you can write a YAML specification.

In that file, you will specify an object called Migration. This object will define the scope of the applications to move and decide whether to automatically start the applications.

Paste this to a file named migration.yaml.

apiVersion: stork.libopenstorage.org/v1alpha1
kind: Migration
metadata:
  name: mysqlmigration
  namespace: migrationnamespace
spec:
  # This should be the name of the cluster pair created above
  clusterPair: remotecluster
  # If set to false this will migrate only the Portworx volumes. No PVCs, apps, etc will be migrated
  includeResources: true
  # If set to false, the deployments and stateful set replicas will be set to 0 on the destination.
  # There will be an annotation with "stork.openstorage.org/migrationReplicas" on the destinationto store the replica count from the source.
  startApplications: true
  # List of namespaces to migrate
  namespaces:
  - migrationnamespace

Next, you can invoke this migration manually from the command line:

kubectl apply -f migration.yaml
Migration mysqlmigration created successfully

or automate it through storkctl:

storkctl create migration mysqlmigration --clusterPair remotecluster --namespaces migrationnamespace --includeResources --startApplications -n migrationnamespace
Migration mysqlmigration created successfully

Migration scope

Currently, you can only migrate namespaces in which the object is created. You can also designate one namespace as an admin namespace. This will allow an admin who has access to that namespace to migrate any namespace from the source cluster. Instructions for setting this admin namespace to stork can be found here

Monitoring a migration

Once the migration has been started using the above commands, you can check the status using storkctl:

storkctl get migration -n migrationnamespace

First, you should see something like this:

NAME            CLUSTERPAIR     STAGE     STATUS       VOLUMES   RESOURCES   CREATED
mysqlmigration  remotecluster   Volumes   InProgress   0/1       0/0         26 Oct 18 20:04 UTC

If the migration is successful, the Stage will go from Volumes→ Application→Final.

Here is how the output of a successful migration would look like:

NAME            CLUSTERPAIR     STAGE     STATUS       VOLUMES   RESOURCES   CREATED
mysqlmigration  remotecluster   Final     Successful   1/1       3/3         26 Oct 18 20:04 UTC

Troubleshooting

If there is a failure or you want more information about what resources were migrated you can describe the migration object using kubectl:

kubectl describe migration mysqlmigration
Name:         mysqlmigration
Namespace:    migrationnamespace
Labels:       <none>
Annotations:  <none>
API Version:  stork.libopenstorage.org/v1alpha1
Kind:         Migration
Metadata:
  Creation Timestamp:  2018-10-26T20:04:19Z
  Generation:          1
  Resource Version:    2148620
  Self Link:           /apis/stork.libopenstorage.org/v1alpha1/migrations/ctlmigration3
  UID:                 be63bf72-d95a-11e8-ba98-0214683e8447
Spec:
  Cluster Pair:       remotecluster
  Include Resources:  true
  Namespaces:
      migrationnamespace
  Selectors:           <nil>
  Start Applications:  true
Status:
  Resources:
    Group:      core
    Kind:       PersistentVolume
    Name:       pvc-34bacd62-d7ee-11e8-ba98-0214683e8447
    Namespace:
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       PersistentVolumeClaim
    Name:       mysql-data
    Namespace:  mysql
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      apps
    Kind:       Deployment
    Name:       mysql
    Namespace:  mysql
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
  Stage:        Final
  Status:       Successful
  Volumes:
    Namespace:                mysql
    Persistent Volume Claim:  mysql-data
    Reason:                   Migration successful for volume
    Status:                   Successful
    Volume:                   pvc-34bacd62-d7ee-11e8-ba98-0214683e8447
Events:
  Type    Reason      Age    From   Message
  ----    ------      ----   ----   -------
  Normal  Successful  2m42s  stork  Volume pvc-34bacd62-d7ee-11e8-ba98-0214683e8447 migrated successfully
  Normal  Successful  2m39s  stork  /v1, Kind=PersistentVolume /pvc-34bacd62-d7ee-11e8-ba98-0214683e8447: Resource migrated successfully
  Normal  Successful  2m39s  stork  /v1, Kind=PersistentVolumeClaim mysql/mysql-data: Resource migrated successfully
  Normal  Successful  2m39s  stork  apps/v1, Kind=Deployment mysql/mysql: Resource migrated successfully

Pre and Post Exec rules

Similar to snapshots, a PreExec and PostExec rule can be specified when creating a Migration object. This will result in the PreExec rule being run before the migration is triggered and the PostExec rule to be run after the Migration has been triggered. If the rules do not exist, the Migration will log an event and will stop.

If the PreExec rule fails for any reason, it will log an event against the object and retry. The Migration will not be marked as failed.

If the PostExec rule fails for any reason, it will log an event and mark the Migration as failed. It will also try to cancel the migration that was started from the underlying storage driver.

As an example, to add pre and post rules to our migration, we could edit our migration.yaml file like this:

apiVersion: stork.libopenstorage.org/v1alpha1
kind: Migration
metadata:
  name: mysqlmigration
  namespace: mysql
spec:
  clusterPair: remotecluster
  includeResources: true
  startApplications: true
  preExecRule: mysql-pre-rule
  postExecRule: mysql-post-rule
  namespaces:
  - mysql

Advanced Operations



Last edited: Wednesday, Oct 9, 2019