Rejoin a decommissioned Portworx node back to the cluster in airgapped bare metal

This document provides instructions for rejoining a previously decommissioned node with its original cluster.

Ensure the node is decommissioned from Portworx

If the node was previously part of a Portworx cluster, you must first ensure you followed steps to decommission it from the cluster. The Decommission a Node page has detailed steps for this.

Ensure the Portworx services are no longer running

You must ensure that the Portworx services are completely disabled from your system. You can check the status of the portworx systemd service using the following systemctl command:

sudo systemctl status portworx

If the systemd service is still running, enter the following commands to stop and disable it:

sudo systemctl stop portworx
sudo systemctl disable portworx
sudo rm -f /etc/systemd/system/portworx*
grep -q '/opt/pwx/oci /opt/pwx/oci' /proc/self/mountinfo && sudo umount /opt/pwx/oci

note

Do not remove /opt/pwx or /etc/pwx at this stage if you want to use pxctl to clean up metadata in the next step. If you've already removed these directories, skip to the Remove metadata using wipefs section.

Clean up Portworx metadata on the node

Portworx also stores local metadata on the node to fingerprint its identity. The method you used to decommission the node determines the steps you must follow to clean up local Portworx metadata.

Remove metadata using pxctl

If pxctl is still on your node, enter the following pxctl service command to remove local metadata using pxctl:

pxctl service node-wipe --all

After running the above command, clean up the Portworx installation directories:

sudo umount /opt/pwx/oci
sudo rm -rf /etc/pwx /opt/pwx

Remove metadata using wipefs

If pxctl has already been removed from your node (or you removed /opt/pwx in the previous step), you must manually find and wipe Portworx disks:

1. Enter the following blkid and grep command to list the disks that Portworx was using:

   blkid | grep pxpool

2. For each of the disks output in step one, enter the wipefs command with the -af options and the <disk-name> to wipe the Portworx metadata:

   wipefs -af <disk-name>

   Example

   wipefs -af /dev/sdf

Restart the Portworx pod on your node by cleaning up labels

Once you have completely decommissioned your node and wiped it of Portworx data, you're ready to re-add it to your cluster. Enter the following kubectl label nodes commands, replacing <node> with the Kubernetes node name of this node:

kubectl label node <node-name> px/service=restart --overwrite
kubectl label node <node-name> px/enabled=true --overwrite 

Once you've entered these commands, Portworx will start on this node and rejoin the cluster as a new node.