Skip to main content
Version: 3.1

Set up a witness node for synchronous DR in Tanzu

In a Portworx cluster, quorum refers to the minimum number of active storage nodes necessary to maintain cluster operation. If at least half of the nodes are offline, the cluster loses quorum causing all operations to stop and Portworx does not process any IOs.

In a Synchronous DR setup, a single Portworx cluster spans two Kubernetes clusters, each within a Metro Area Network. The same quorum principles apply here, with all storage nodes from both source and destination data centers contributing to quorum. Portworx quorum can be lost in the event of a disaster.

To solve the quorum issue, you can deploy a witness site that is used as the quorum tie-breaker when there is a network partition or when a data center goes offline. The witness node is a single virtual machine and a special Portworx storageless node that participates in quorum, but does not store any data. Typically situated in a third data center, the witness node ensures quorum integrity.

Prerequisites​

Setup a witness node​

Perform the following to set up a witness node:

  1. Check your Portworx Enterprise version by running the following command on your source and destination clusters (both should have the same version):

    kubectl get pods -A -o jsonpath="{.items[*].spec.containers[*].image}" | xargs -n1 | sort -u | grep oci-monitor

  2. Download the witness-install.sh script file on a designated VM.

  3. Install the witness node on a single storageless Portworx node on the designated VM. You need to specify the same Portworx Enterprise version that you retrieved in Step 1 along with the external etcd endpoints, as shown in the following example:

    sh witness-install.sh --cluster-id=px-cluster  \
     --etcd="etcd:http://<your-etcd-endpoint1>:2379,etcd:http://<your-etcd-endpoint2>:2379,etcd:http://<your-etcd-endpoint3>:2379"  \
     --docker-image=portworx/px-enterprise:<your-px-version>

  4. Verify Portworx status on the witness node:

    pxctl status

    The witness-install.sh script can take a couple of minutes to complete, as shown in the following example output:

    Status: PX is operational
    Telemetry: Disabled or Unhealthy
    Metering: Disabled or Unhealthy
    License: Trial (expires in 30 days)
    ..
    ....

    You will see PX is operational and once the script is successfully completed, you can quit the script by entering ctrl + c. Note that the witness node requires a valid Portworx license. To check the status of your license, use the pxctl license list command.

Last updated on