Skip to main content
Version: 3.1

Adding storage to existing Portworx Cluster Nodes

Deprecation notice

Support for Portworx on the following orchestrators will be deprecated:

  • Docker Swarm and Docker Standalone
  • Nomad

For installation instructions on other platforms, refer to the Platform guides section.

note

This document presents the non-Kubernetes method of scaling your Portworx cluster. Please refer to the Scale or Restrict page if you are running Portworx on Kubernetes.

Adding Storage to existing Portworx Cluster Nodes​

This section illustrates how to add a new node to a Portworx cluster and how to add additional storage to the Portworx Cluster once a new node is added.

Display current cluster status​

pxctl status
Status: PX is operational
Node ID: xxxxxxxx-xxxx-xxxx-xxxx-3e2b01cd0bc3
	IP: X.X.X.0
 	Local Storage Pool: 2 pools
	Pool	IO_Priority	Size	Used	Status	Zone	Region
	0	LOW		200 GiB	1.0 GiB	Online	default	default
	1	LOW		120 GiB	1.0 GiB	Online	default	default
	Local Storage Devices: 2 devices
	Device	Path				Media Type		SizLast-Scan
	0:1	/dev/mapper/volume-27dbb728	STORAGE_MEDIUM_SSD	200 GiB		08 Jan 17 16:54 UTC
	1:1	/dev/mapper/volume-0a31ef46	STORAGE_MEDIUM_SSD	120 GiB		08 Jan 17 16:54 UTC
	total					-			320 GiB
Cluster Summary
	Cluster ID: xxxxxxxx-xxxx-xxxx-xxxx-0242ac110002
	Node IP: X.X.X.197 - Capacity: 2.0 GiB/320 GiB Online (This node)
	Node IP: 10.99.119.1 - Capacity: 1.2 GiB/100 GiB Online
	Node IP: 10.99.117.129 - Capacity: 1.2 GiB/100 GiB Online
Global Storage Pool
	Total Used    	:  4.3 GiB
	Total Capacity	:  520 GiB

The above cluster has three nodes and 520GiB of total capacity.

Add a new node to cluster​

Below is an example of how to run Portworx in a new node so it joins an existing cluster. Note how docker run command is invoked with a cluster token token-xxxxxxxx-xxxx-xxxx-xxxx-0242ac110002 that has a token- prefix to the cluster ID to which we want to add the new node.

docker run --restart=always --name px-enterprise -d --net=host --privileged=true -v /run/docker/plugins:/run/docker/plugins -v /var/lib/osd:/var/lib/osd:shared -v /dev:/dev -v /etc/pwx:/etc/pwx -v /opt/pwx/bin:/export_bin:shared -v /var/run/docker.sock:/var/run/docker.sock -v /mnt:/mnt:shared -v /var/cores:/var/cores -v /usr/src:/usr/src portworx/px-enterprise -m team0:0 -d team0

Here is how the cluster would look like after a new node is added without any storage:

pxctl status
Status: PX is operational
Node ID: xxxxxxxx-xxxx-xxxx-xxxx-c9d0eb597668
	IP: X.X.X.0
 	Local Storage Pool: 0 pool
	Pool	IO_Priority	Size	Used	Status	Zone	Region
	No storage pool
	Local Storage Devices: 0 device
	Device	Path	Media Type	Size		Last-Scan
	No storage device
	total		-	0 B
Cluster Summary
	Cluster ID: xxxxxxxx-xxxx-xxxx-xxxx-0242ac110002
	Node IP: 10.99.119.1 - Capacity: 1.2 GiB/100 GiB Online
	Node IP: 10.99.117.129 - Capacity: 1.2 GiB/100 GiB Online
	Node IP: X.X.X.197 - Capacity: 2.0 GiB/320 GiB Online
	Node IP: X.X.X.185 - Capacity: 0 B/0 B Online (This node)
Global Storage Pool
	Total Used    	:  4.3 GiB
	Total Capacity	:  520 GiB

Note how the capacity of the cluster has remained unchanged.

Add more storage to the new node​

Added another 100G of storage to this node and the device is seen as /dev/dm-1:

multipath -ll
volume-a9e55549 (360014055671ce0d20184a619c27b31d0) dm-1   ,IBLOCK
size=100G features='0' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 2:0:0:0 sdb 8:16 active ready running
  `- 3:0:0:0 sdc 8:32 active ready running

Add the new drive to cluster to increase the storage​

pxctl service drive add --drive /dev/dm-1 --operation start
Adding device  /dev/dm-1 ...
"Drive add done: Storage rebalance is in progress"

Rebalance the storage pool​

note

Pool rebalance is a must. It spreads data across all available drives in the pool.

Check the rebalance status and wait for completion:

pxctl service drive add --drive /dev/dm-1 --operation status
"Drive add: Storage rebalance running: 1 out of about 9 chunks balanced (2 considered),  89% left"
pxctl service drive add --drive /dev/dm-1 --operation status
"Drive add: Storage rebalance complete"

In case drive add operation did not start a rebalance, start it manually. For e.g., if the drive was added to pool 0:

pxctl service drive rebalance --poolID 0 --operation start
Done: "Pool 0: Balance is running"

Check the rebalance status and wait for completion:

pxctl service drive rebalance --poolID 0 --operation status
Done: "Pool 0: Balance is not running"

Check cluster status​

As seen below, the 100G of additional capacity is available with total capacity of the cluster going to 620GB

pxctl status
Status: PX is operational
Node ID: xxxxxxxx-xxxx-xxxx-xxxx-c9d0eb597668
	IP: X.X.X.0
 	Local Storage Pool: 1 pool
	Pool	IO_Priority	Size	Used	Status	Zone	Region
	0	LOW		100 GiB	1.0 GiB	Online	default	default
	Local Storage Devices: 1 device
	Device	Path				Media Type		Size	Last-Scan
	0:1	/dev/mapper/volume-a9e55549	STORAGE_MEDIUM_SSD	100 GiB08 Jan 17 21:46 UTC
	total					-			100 GiB
Cluster Summary
	Cluster ID: xxxxxxxx-xxxx-xxxx-xxxx-0242ac110002
	Node IP: 10.99.119.1 - Capacity: 1.2 GiB/100 GiB Online
	Node IP: X.X.X.197 - Capacity: 2.0 GiB/320 GiB Online
	Node IP: X.X.X.185 - Capacity: 0 B/100 GiB Online (This node)
	Node IP: 10.99.117.129 - Capacity: 1.2 GiB/100 GiB Online
Global Storage Pool
	Total Used    	:  4.3 GiB
	Total Capacity	:  620 GiB
Last updated on