Place replicas within the same domain

Summary and Key concepts

Summary

This article explains how to manage the placement of Portworx volume replicas across cluster domains in a Synchronous Disaster Recovery (DR) environment using the Metro DR domain protection flag, which enforces cross-domain distribution by default. It includes steps for creating volume replicas within the same cluster domain if needed, such as labeling volumes with disable-domain-protection or using a custom Volume Placement Strategy (VPS) for specific applications. The article guides users on defining and applying custom storage classes to control domain protection settings, including creating Persistent Volume Claims (PVCs) for volumes with specific domain placement.

Kubernetes Concepts

• StorageClass: Defines storage parameters, including custom settings for replica placement strategies, allowing volumes to follow specified domain protection or disabling domain protection.
• PersistentVolumeClaim (PVC): Used to request storage resources, here with customized domain-protection settings for volume replica placement within the Portworx cluster.

Portworx Concepts

• VolumePlacementStrategy (VPS): A Portworx custom resource that allows users to specify where volume replicas are placed, enabling finer control over replica locality across domains.

• pxctl: Portworx command-line tool used here to check and configure Metro DR domain protection settings within a Portworx cluster.

Once your Portworx cluster is operational, the replica 2 volumes will distribute their replicas across the two cluster domains. You can control this behavior using the Metro DR domain protection flag, which is enabled by default.

You can run the following command to check if the protection flag is enabled in your setup:

PX_POD=$(kubectl get pods -l name=portworx -n <portworx-namespace> -o jsonpath='{.items[0].metadata.name}') \ 
kubectl exec $PX_POD -n <portworx-namespace> --  /opt/pwx/bin/pxctl cluster options list | grep Metro

If you want the volume replica of a specific volume (for example, monitoring data volumes) to be created within the same cluster domain, label the volume with disable-domain-protection. This label disables the Metro DR domain protection flag at the individual volume level, rather than at the cluster level.

caution

The volumes with the disable-domain-protection label will not be protected by Synchronous DR. In the event of a disaster, you might lose the data associated with these volumes.

To force both the replica provision in the same cluster domain use the VPS. If you do not want to enforce this behavior and the goal is to simply relax the Metro DR domain protection, then you can skip step 1 and use only disable-domain-protection: "true" in the StorageCluster spec.

1. Create a custom volume placement strategy for replicaAffinity, so that volume replicas are always in the same cluster domain:

   apiVersion: portworx.io/v1beta2
   kind: VolumePlacementStrategy
   metadata:
      name: vps-domain-filter
   spec:
      replicaAffinity:
      - matchExpressions:
        - key: domain
          operator: In
          values:
          - <domain-name>

   Replace <domain-name> with the domain name where you want the replicas to be placed.

2. Create the following StorageClass:

   kind: StorageClass
   apiVersion: storage.k8s.io/v1
   metadata:
       name: disable-domain-protection-sc
   provisioner: pxd.portworx.com
   parameters:
      repl: "2"
      disable-domain-protection : "true"
      placement_strategy: "vps-domain-filter"
   allowVolumeExpansion: true

   If you are not using the VPS, then remove placement_strategy: "vps-domain-filter" from the above spec.

3. Save and apply the above spec:

   kubectl apply -f <your-storageclass>.yaml

4. Create a PVC which references the StorageClass you created above, specifying the StorageClass:

   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
      name: <application>-pvc
   spec:
      storageClassName: disable-domain-protection-sc
      accessModes:
        - ReadWriteOnce
      resources:
      requests:
        storage: 2Gi

5. Save and apply your PVC with the kubectl apply command:

kubectl apply -f <your-storageclass>.yaml

The above PVC will have its replicas placed within the same cluster domain.