Skip to main content

Decommission a Portworx node

This page provides the steps to safely decommission a Portworx node from your Kubernetes cluster. The process includes migrating application pods, removing Portworx from the cluster, and cleaning up associated metadata.

Migrate application pods​

Before decommissioning a Portworx node, migrate any applications running on it. Follow these steps:

  1. Cordon the node to prevent new pods from being scheduled:

    kubectl cordon <node>

    This ensures no new pods are scheduled on the node being decommissioned.

  2. Delete the pods running on the node to trigger rescheduling by Kubernetes on another node. Pods managed by a Deployment or StatefulSet will be automatically rescheduled:

    kubectl delete pod <pod-name> -n <portworx>

    Ensure that the rescheduled pods are running on other nodes before proceeding.

Remove Portworx node from the cluster​

Follow the instructions to remove a node from the cluster.

To remove an online Portworx node, you can run the commands either on the node itself or from another node.

  1. Check the Portworx cluster status and identify the node to be decommissioned.

    pxctl status
    Status: PX is operational
    Node ID: xxxxxxxx-xxxx-xxxx-xxxx-0027f6bbcbd1
            IP: X.X.X.0
            Local Storage Pool: 1 pool
            POOL    IO_PRIORITY     SIZE    USED    STATUS  ZONE    REGION
            0       LOW             64 GiB  11 GiB  Online  c       us-east-1
            Local Storage Devices: 1 device
            Device  Path            Media Type              Size            Last-Scan
            0:1     /dev/xvdf       STORAGE_MEDIUM_SSD      64 GiB          25 Feb 17 21:13 UTC
            total                   -                       64 GiB
    Cluster Summary
            Cluster ID: xxxxxxxx-xxxx-xxxx-xxxx-d4a612b74cc3
            IP              ID                                      Used    Capacity        Status
            172.31.40.38    xxxxxxxx-xxxx-xxxx-xxxx-2bc112f5f131    11 GiB  64 GiB          Online
            172.31.37.211   xxxxxxxx-xxxx-xxxx-xxxx-a85e0514ae8b    11 GiB  64 GiB          Online
            172.31.35.130   xxxxxxxx-xxxx-xxxx-xxxx-893373631483    11 GiB  64 GiB          Online
            172.31.45.106   xxxxxxxx-xxxx-xxxx-xxxx-2eeddcd64d51    11 GiB  64 GiB          Online
            172.31.45.56    xxxxxxxx-xxxx-xxxx-xxxx-ec8e1420e645    11 GiB  64 GiB          Online
            172.31.46.119   xxxxxxxx-xxxx-xxxx-xxxx-0027f6bbcbd1    11 GiB  64 GiB          Online (This node)
            172.31.39.201   xxxxxxxx-xxxx-xxxx-xxxx-936b1b58aa24    11 GiB  64 GiB          Online
            172.31.33.151   xxxxxxxx-xxxx-xxxx-xxxx-41e70a72eafd    11 GiB  64 GiB          Online
            172.31.33.252   xxxxxxxx-xxxx-xxxx-xxxx-428e727eb6b8    11 GiB  64 GiB          Online
    Global Storage Pool
            Total Used      :  99 GiB
            Total Capacity  :  576 GiB

  2. Log in to the node and place it in maintenance mode to avoid any disruptions to ongoing operations while decommissioning.

    pxctl service maintenance --enter
    This is a disruptive operation, PX will restart in maintenance mode.
    Are you sure you want to proceed? (Y/N): y
    Entered maintenance mode.

  3. (Optional) Add px/enabled=remove label to explicitly mark the node for removal, helping automated systems (like Terraform or Kubernetes Operators) identify and manage it. It ensures offline nodes are not skipped, prevents errors, and improves visibility during decommissioning.

    kubectl label nodes <node-id> px/enabled=remove --overwrite

  4. Remove the node from the cluster by running the following command from a different node or the same node:

    pxctl cluster delete <node-id>
    Node xxxxxxxx-xxxx-xxxx-xxxx-0027f6bbcbd1 successfully deleted.

  5. After removing the node, run the pxctl service node-wipe command to wipe the Portworx metadata from the node.

    pxctl service node-wipe --all

Uncordon the node​

note

This step is not required if you want to remove a node completely from the Kubernetes cluster. Use the kubectl delete node command to delete the node from the cluster.

If you want to reuse the node for application deployment without running Portworx, uncordon the node:

  1. Uncordon the node to allow scheduling of applications.

    kubectl uncordon <node-id>

Automatic decommissioning of offline nodes​

If a node is taken offline or destroyed, the cluster will retain the node and mark it as offline. To remove such nodes, you need to run pxctl cluster delete commands. This process can become laborious with an increasing number of offline nodes or a higher frequency of nodes being added and taken down.

To address this, Portworx automatically waits for a grace period of 48 hours. After this period, offline nodes are removed from the cluster.

Rejoin a decommisioned Portworx node to the cluster​

If you want to rejoin a decommissioned Portworx node to the cluster (e.g., after maintenance), follow the steps below.

  1. Ensure that Portworx services are not running on the node.

    sudo systemctl status portworx

    If the systemd services are running, run the following commands to stop it:

    sudo systemctl stop portworx
    sudo systemctl disable portworx
    sudo rm -f /etc/systemd/system/portworx*
    grep -q '/opt/pwx/oci /opt/pwx/oci' /proc/self/mountinfo && sudo umount /opt/pwx/oci

  2. run the pxctl service node-wipe command to wipe the Portworx metadata from the node.

    pxctl service node-wipe --all

  3. Add the node to Portworx cluster using the following command:

    kubectl label node <node> px/service=restart --overwrite
    kubectl label node <node> px/enabled=true --overwrite

    Portworx now starts on this node and rejoins the cluster as a new node.

Last updated on