Prepare your Portworx cluster


In the Metro DR deployment model, you need to have a single Portworx cluster that spans across your source and destination Kubernetes clusters. Follow the instructions on this page to achieve the same.

Install Portworx

Follow the instructions on this page to deploy a single Portworx cluster that spans across multiple Kubernetes clusters. You should specify the same KVDB endpoints and a cluster name for the Portworx cluster on your source and destination clusters.

Generate a Portworx spec

  1. Navigate to PX-Central to log in or create an account.

  2. On the Basic page, select the Your etcd details option and enter the following KVDB endpoints, one at a time using the + symbol, then click Next:

    • http://10.13.28.137:2379
    • http://10.13.28.138:2379
    • http://10.13.28.139:2379
  3. On the Storage page, select cloud as your environment and choose your cloud provider. Specify a number in the Max storage nodes per availability zone (Optional) field to match the total number of Kubernetes storage nodes in your stretch cluster, and click Next.

    NOTE: Specifying 0 in the Max storage nodes per availability zone (Optional) field will keep the number of storage nodes unbounded in a cluster. Every node that joins the cluster will be a storage node. This value should be used with great caution.
  4. Choose your network and click Next.

  5. From the Customize page, specify your cluster name prefix in Advanced Settings. Click Finish to generate the spec.

After generating the spec file, copy it to the source and destination clusters. You must add domain names to each of these spec files. The domain names must be different for each Kubernetes cluster. You can use several methods to specify a domain name for your clusters, such as your cloud providers’ zone names (for example, us-east-1a and us-east-1b) or the datacenter names (for example, datacenter1 and datacenter2) that your clusters are using. A cluster domain identifies a subset of nodes from the stretch Portworx cluster that are part of the same failure domain. Each Kubernetes cluster and its nodes form a cluster domain.

Before applying the specs to you clusters, ensure that both the spec files have the same metadata:name and KVDB endpoints, but different domain names.

Apply the spec on your source cluster

Once you have copied the spec to your source cluster, modify it to add the source cluster domain name.

  1. Specify a domain name using the portworx.io/misc-args annotation with the -cluster_domain argument. The following example shows how you can specify your source cluster domain name using your cloud providers’ zone name (for example, us-east-1a):

    apiVersion: core.libopenstorage.org/v1
    kind: StorageCluster
    metadata:
      annotations:
        portworx.io/misc-args: "-cluster_domain us-east-1a"
  2. Apply the edited spec:

    kubectl apply -f <your-px-source-spec>.yaml

Apply the spec on your destination cluster

Similarly, after you have copied the spec to your destination cluster, modify it to add the destination cluster domain name.

  1. Specify the domains using the -cluster_domain argument with the portworx.io/misc-args annotation. You should use the same method as your source cluster for specifying the domain name of your destination cluster. The following example shows how you can specify the domain name using us-east-1b:

    apiVersion: core.libopenstorage.org/v1
    kind: StorageCluster
    metadata:
      annotations:
        portworx.io/misc-args: "-cluster_domain us-east-1b"
  2. Apply the edited spec:

    kubectl apply -f <your-px-destination-spec>.yaml

A single Portworx cluster will be spanning across both Kubernetes clusters since the metadata:name and KVDB endpoints are the same.

Install storkctl

storkctl is a command-line tool for interacting with a set of scheduler extensions. Install storkctl on both Kubernetes clusters after installing Portworx.

NOTE: The examples below use the kube-system namespace, you should update this to the correct namespace for your environment.

Perform the following steps to download storkctl from the Stork pod:

  • Linux:

    STORK_POD=$(kubectl get pods -n <namespace> -l name=stork -o jsonpath='{.items[0].metadata.name}') &&
    kubectl cp -n <namespace> $STORK_POD:/storkctl/linux/storkctl ./storkctl
    sudo mv storkctl /usr/local/bin &&
    sudo chmod +x /usr/local/bin/storkctl
  • OS X:

    STORK_POD=$(kubectl get pods -n <namespace> -l name=stork -o jsonpath='{.items[0].metadata.name}') &&
    kubectl cp -n <namespace> $STORK_POD:/storkctl/darwin/storkctl ./storkctl
    sudo mv storkctl /usr/local/bin &&
    sudo chmod +x /usr/local/bin/storkctl
  • Windows:

    1. Copy storkctl.exe from the stork pod:

      STORK_POD=$(kubectl get pods -n <namespace> -l name=stork -o jsonpath='{.items[0].metadata.name}') &&
      kubectl cp -n <namespace> $STORK_POD:/storkctl/windows/storkctl.exe ./storkctl.exe
    2. Move storkctl.exe to a directory in your PATH.

Check the cluster domain status

When Stork is deployed along with Portworx, it automatically creates a ClusterDomainStatus object on your source and destination Kubernetes clusters.

  1. Run the following commands on each of your Kubernetes clusters to get the current status of the ClusterDomainStatus object:

    storkctl get clusterdomainsstatus
    NAME                            LOCAL-DOMAIN   ACTIVE                                     INACTIVE   CREATED
    px-dr-cluster                   us-east-1a     us-east-1a (InSync), us-east-1b (InSync)              29 Nov 22 22:09 UTC
  2. Run the following command to get details of the ClusterDomainStatus object:

    kubectl describe clusterdomainsstatus
    Name:         px-dr-cluster
    Namespace:    
    Labels:       <none>
    Annotations:  <none>
    API Version:  stork.libopenstorage.org/v1alpha1
    Kind:         ClusterDomainsStatus
    Metadata:
    Creation Timestamp:  2022-11-28T23:13:32Z
    Generation:          3
    Managed Fields:
    API Version:  stork.libopenstorage.org/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .: 
        f:clusterDomainInfos:
        f:localDomain:
    Manager:         stork
    Operation:       Update
    Time:            2022-11-28T23:13:32Z
    Resource Version:  3383
    UID:               xxxxxxxx-2e81-4cbf-b4da-4b9c261793fd
    Status:
    Cluster Domain Infos:
    Name:         us-east-1b
    State:        Active
    Sync Status:  InSync
    Name:         us-east-1a
    State:        Active
    Sync Status:  InSync
    Local Domain:   us-east-1a
    Events:           <none>

    You can see that the Status field shows that your Portworx stretch cluster is installed and active.

Setup a witness node

Perform the following to set up a witness node:

  1. Check your Portworx Enterprise version by running the following command on your source and destination clusters (both should have the same version):

    kubectl get pods -A -o jsonpath="{.items[*].spec.containers[*].image}" | xargs -n1 | sort -u | grep oci-monitor
  2. Download the witness-install.sh script file on a designated VM.

  3. Change the permissions for the downloaded file to -rwxr-xr-x:

    chmod +x ./witness-install.sh
  4. Update the Portworx version in the PX_DOCKER_IMAGE=portworx/px-enterprise:2.XX.X field of the file to match the version that is retrieved in Step 1 of this section. Check the following and save the file:

    • The --cluster_domain argument is set to witness, indicating that this is a witness node.
    • The clusterID and the etcd details are same as they have been provided to the two other Portworx installations done in the Kubernetes clusters.
  5. Verify if the docker engine service is running. This service is required in order to successfully run the witness node script:

    systemctl status docker
    docker.service - Docker Application Container Engine
    Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
    Active: active (running) since Tue 2022-11-29 18:02:09 EST; 18h ago
     Docs: https://docs.docker.com
    Main PID: 1356 (dockerd)
    Tasks: 10
    Memory: 71.8M
    CGroup: /system.slice/docker.service
           └─1356 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
  6. Install the witness node on a single storageless Portworx node on the designated VM. You need to specify the same Portworx Enterprise version that you retrieved in Step 1 along with the external etcd endpoints, as shown in the following example:

    ./witness-install.sh --cluster-id=px-cluster  \
     --etcd="etcd:http://10.13.28.137:2379,etcd:http://10.13.28.138:2379,etcd:http://10.13.28.139:2379"  \
     --docker-image=portworx/px-enterprise:<your-px-version>
  7. Verify Portworx status on the witness node:

    /opt/pwx/bin/pxctl --color status

    The witness-install.sh script can take a couple of minutes to complete, as shown in the following example output:

      Status: PX is operational
      Telemetry: Disabled or Unhealthy
      Metering: Disabled or Unhealthy
     License: Trial (expires in 30 days)
      ..
      ....

    You will see PX is operational and once the script is successfully completed, you can quit the script by entering ctrl + c. Note that the witness node requires a valid Portworx license. To check the status of your license, use the /opt/pwx/bin/pxctl --color license list command.

    NOTE:



Last edited: Wednesday, Feb 1, 2023