Manage storage pool capacity for FlashArray
As your cluster usage increases and the data on your storage pools grows, you may start to run out of capacity. Portworx periodically measures and saves storage pool space usage statistics, which can be queried via the appropriate metrics or the CLI tool.
This page provides information on how Portworx alerts you when a storage pool reaches its maximum capacity and the consequences of taking no action. It also explains how to recover your pool if it has reached its maximum capacity.
Alerts and impacts when a pool reaches maximum capacity
To manage your pools' storage capacity, Portworx triggers alerts and takes necessary actions to avoid disruption.
-
When a storage pool reaches 80% of its capacity, Portworx triggers an alert that the storage pool has high space usage.
-
When a storage pool reaches 90% of its capacity, Portworx takes the following actions:
- Sets the pool as full and the node as
StorageDown
. - Rejects all active IOs on the pool with the
ENOSPC
error. - Sets mountpoints as read-only, and write operations will not be allowed.
- Sets the pool as full and the node as
To avoid disruption, you must expand your storage pool capacity by adding drives or increasing disk size. For more details, refer to the Expand your storage pool size section.
Implications of a full pool
If, for some reason, you could not expand your storage pool, it can lead to resource allocation issues. The following table explains the implications of a full pool in different scenarios:
Volume type | Node health | Implications |
---|---|---|
repl 1 | All nodes are marked full |
|
repl 2 or 3 | All nodes are marked full |
|
repl 2 or 3 | Some nodes are marked full | Read operations are allowed as long as the volume is in clean state. |
If all the nodes with volume replication are marked as full, the volume will become unavailable, resulting in no read or write operations.
Revive your storage pool
You can restore a pool by deleting data or volume, or replication volume. Choose one of the following methods to restore your storage pool.
-
Delete volume replicas:
pxctl volume ha-update --repl <current-HA-level -1> --node <node-ID> <volume-name>
Example:
pxctl volume ha-update --repl -1 --node node1 vol1
This command will decrease the number of replicas of the volume on the specified node.
-
Delete data or volume:
pxctl volume delete <vol-name>
- Use
kubectl
to cordon the affected nodes. Run the following command to temporarily increase the free space threshold value and bring the pools online:pxctl cluster options update --free-space-threshold-gb <threshold-value>