Automatically expand Portworx storage pools with Autopilot in IKS
Summary and Key concepts
Summary:
This article explains how to use Portworx Autopilot to automatically expand storage pools when they begin running out of space. Autopilot monitors storage pool metrics (e.g., via Prometheus) and triggers resizing actions when high usage conditions are detected. The example provided demonstrates how to create an Autopilot rule that resizes a storage pool by 50% when its available capacity drops below 50%, up to a maximum of 400GiB. The article includes YAML specs for creating a PostgreSQL application with persistent volumes, configuring storage pools, and applying the Autopilot rule. It also provides monitoring instructions for tracking the rule's execution using Kubernetes events.
Kubernetes Concepts:
- PersistentVolumeClaim (PVC): A request for storage in Kubernetes. The example includes a PostgreSQL application with PVCs to demonstrate Autopilot's pool expansion feature.
- StorageClass: Specifies how storage is provisioned in Kubernetes. The article includes a Portworx storage class definition for PVC expansion.
Portworx Concepts:
- Autopilot: A feature that automates the expansion of storage pools and PVCs based on predefined rules and metrics.
- Storage Pool: A Portworx construct representing a collection of storage resources, which Autopilot can scale or rebalance.
You can use Autopilot to expand Portworx storage pools automatically when they begin to run out of space. Autopilot monitors the metrics in your cluster (e.g., via Prometheus) and detects high usage conditions. Once high usage conditions occur, Autopilot communicates with Portworx to resize the pool.
Prerequisites
- Portworx cloud drives: Your Portworx installation must use one of the supported cloud drives where Portworx provisions the backing drives using the cloud provider
- Autopilot version: 1.2.13 and above
Example spec
You can use the auto
scale type to automatically expand the Portworx storage pool. You can also specify the add-drive
or resize-drive
scale type based on your specific use case.
In the following example, Autopilot rule uses the resize-drive
scale type to expand a 250GiB Portworx storage pool composed of a single 250GiB drive by 50%:
apiVersion: autopilot.libopenstorage.org/v1alpha1
kind: AutopilotRule
metadata:
name: pool-expand
spec:
enforcement: required
##### conditions are the symptoms to evaluate. All conditions are AND'ed
conditions:
expressions:
# pool available capacity less than 50%
- key: "100 * ( px_pool_stats_available_bytes/ px_pool_stats_total_bytes)"
operator: Lt
values:
- "50"
# pool total capacity should not exceed 2TB
- key: "px_pool_stats_total_bytes/(1024*1024*1024)"
operator: Lt
values:
- "2000"
##### action to perform when condition is true
actions:
- name: "openstorage.io.action.storagepool/expand"
params:
# resize pool by scalepercentage of current size
scalepercentage: "50"
# when scaling, resize disks in the pool
scaletype: "resize-drive"
Consider the key sections in this spec: conditions
and actions
.
The conditions
section establishes threshold criteria dictating when the rule must perform its action. In this example, that criteria contains 2 formulas:
100 * ( px_pool_stats_available_bytes/ px_pool_stats_total_bytes)
gives the pool available percentage and theLt
operator puts a condition that pool available capacity percentage should be lower 50%.px_pool_stats_total_bytes/(1024*1024*1024)
gives the total pool capacity in GiB, and theLt
operator to caps it to 400GiB.`
Conditions are combined using AND logic, requiring all conditions to be true for the rule to trigger.
The actions
section specifies what action Portworx performs when the conditions are met. Action parameters modify action behavior, and different actions contain different action parameters.
When you expand your storage pool using the add-drive
scale type, Autopilot increases your pool storage capacity by adding new drives to the pool based on the scale percentage. The new drives will match the capacity of the existing ones because the storage pool consists of drives of equal size. For example, if your current pool consists of 100 GiB drive, with add-drive
scale type, a 100 GiB drive is added to your pool, thus expanding it by 100%. In such cases, scale percentage represents the minimum percentage by which Portworx will scale up the pool.
Define and add Autopilot rule
Perform the following steps to deploy the above example.
Create application and PVC specs
The specs below create an application that writes 300 GiB of data to a 400 GiB volume. If your Storage pools are larger than that, you must change these numbers to ensure the capacity condition triggers.
First, create the storage and application spec files:
-
Create
postgres-sc.yaml
and place the following content inside it.##### Portworx storage class
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: postgres-pgbench-sc
provisioner: pxd.portworx.com
parameters:
repl: "2"
allowVolumeExpansion: true -
Create
postgres-vol.yaml
and place the following content inside it.kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pgbench-data
spec:
storageClassName: postgres-pgbench-sc
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 400Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pgbench-state
spec:
storageClassName: postgres-pgbench-sc
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi -
Create
postgres-app.yaml
and place the following content inside it.The application in this example is a PostgreSQL database with a pgbench sidecar. The
SIZE
environment variable in this spec instructs pgbench to write 300GiB of data to the volume. Since the volume is 400GiB in size, Autopilot will resize the storage pool when theconditions
threshold is crossed.apiVersion: apps/v1
kind: Deployment
metadata:
name: pgbench
labels:
app: pgbench
spec:
selector:
matchLabels:
app: pgbench
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
replicas: 1
template:
metadata:
labels:
app: pgbench
spec:
schedulerName: stork
containers:
- image: postgres:9.5
name: postgres
ports:
- containerPort: 5432
env:
- name: POSTGRES_USER
value: pgbench
- name: POSTGRES_PASSWORD
value: superpostgres
- name: PGBENCH_PASSWORD
value: superpostgres
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: pgbenchdb
- name: pgbench
image: portworx/torpedo-pgbench:latest
imagePullPolicy: "Always"
env:
- name: PG_HOST
value: 127.0.0.1
- name: PG_USER
value: pgbench
- name: SIZE
value: "300"
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: pgbenchdb
- mountPath: /pgbench
name: pgbenchstate
volumes:
- name: pgbenchdb
persistentVolumeClaim:
claimName: pgbench-data
- name: pgbenchstate
persistentVolumeClaim:
claimName: pgbench-state
AutopilotRule spec
Once you've created your storage and application specs, you can create an AutopilotRule that controls them.
Create a YAML spec for the autopilot rule named autopilotrule-pool-expand-example.yaml
and place the following content inside it:
apiVersion: autopilot.libopenstorage.org/v1alpha1
kind: AutopilotRule
metadata:
name: pool-expand
spec:
enforcement: required
##### conditions are the symptoms to evaluate. All conditions are AND'ed
conditions:
expressions:
# pool available capacity less than 50%
- key: "100 * ( px_pool_stats_available_bytes/ px_pool_stats_total_bytes)"
operator: Lt
values:
- "50"
# volume total capacity should not exceed 400GiB
- key: "px_pool_stats_total_bytes/(1024*1024*1024)"
operator: Lt
values:
- "400"
##### action to perform when condition is true
actions:
- name: "openstorage.io.action.storagepool/expand"
params:
# resize pool by scalepercentage of current size
scalepercentage: "50"
# when scaling, resize disks in the pool
scaletype: "resize-drive"
Apply specs
Once you've designed your specs, deploy them.
kubectl apply -f autopilotrule-pool-expand-example.yaml
kubectl apply -f postgres-sc.yaml
kubectl apply -f postgres-vol.yaml
kubectl apply -f postgres-app.yaml
Monitor
Observe how the pgbench pod starts filling up the pgbench-data PVCs and, by extension, the underlying Portworx storage pools. As the pool usage exceeds 50%, Autopilot resizes the storage pools.
You can enter the following command to retrieve all the events generated for the pool-expand
rule:
kubectl get events --field-selector involvedObject.kind=AutopilotRule,involvedObject.name=pool-expand --all-namespaces --sort-by .lastTimestamp