Application I/O control in OCP on bare metal
Portworx volumes are created from a common storage pool and share their available IOPS and bandwidth between all other provisioned Portworx volumes. If an application's volumes start consuming too many resources, it can become a "noisy neighbor" and reduce the I/O and network bandwidth available to other volumes in the pool.
The Portworx Application I/O Control feature provides you with a method for controlling an individual Portworx volume’s IOPS or bandwidth usage from within your cluster. You can define quotas through a StorageClass which prevent any applications from using excessive bandwidth. When you need to adjust the bandwidth, you can do so in real-time by updating the values in the StorageClass. The updated configuration will be applied to new volumes created after the change.
In addition to defining parameters in a StorageClass, you can set Application I/O Control values for volumes using the pxctl
command line interface.
By default, this feature is not enabled.
Prerequisites
- Application I/O Control is supported for kernel 4.6 or higher.
- Only the cgroup mount path
/sys/fs/cgroup/blkio
is supported. This is typically set bymount -t cgroup -o blkio none /sys/fs/cgroup/blkio
. Uncommon distributions could have a different cgroup mount path.
Create volumes
-
You can set a maximum rate for IOPS and a maximum rate for bandwidth at the same time. However, you cannot set a read parameter for both IOPS and bandwidth at the same time or a write parameter for both at the same time. For example, setting
--max_iops 1024,off --max_bandwidth off,2
will work, but setting--max_iops 1024,off --max_bandwidth 2,off
will prompt error. -
Once you set Application I/O Control values, you can check alerts or logs for messages indicating that Application I/O Control is not enabled. For example:
IO throttling supported from kernel v4.6, this node has kernel v3.10
indicates that Application I/O Control is not supported. -
IOPS might be misleading due to batching of small blocksize IOs into a larger one before IO reaches the pxd device, especially when using sharedV4 volumes. Bandwidth throttling is more consistent.
Apply using a StorageClass
You can apply Application I/O Control through a StorageClass.
To configure IOPS throttling in a StorageClass, use the following format:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: portworx-io-profile
provisioner: pxd.portworx.com
parameters:
repl: "2"
io_profile: "db_remote"
io_throttle_rd_iops: "1024"
io_throttle_wr_iops: "1024"
To configure bandwidth throttling in a StorageClass, use the following format:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: portworx-io-profile
provisioner: pxd.portworx.com
parameters:
repl: "2"
io_profile: "db_remote"
io_throttle_rd_bw: "10"
io_throttle_wr_bw: "10"
You can also configure read IOPS throttling and write bandwidth throttling, or the inverse. For example:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: portworx-io-profile
provisioner: pxd.portworx.com
parameters:
repl: "2"
io_profile: "db_remote"
io_throttle_rd_iops: "1024"
io_throttle_wr_bw: "10"
The minimum value for rd_bw
and wr_bw
is 1
, and the minimum value for rd_iops
and wr_iops
is 256
.
Create using pxctl
When you create volumes using pxctl
, you can specify the --max_bandwidth
and --max_iops
flags to restrict the storage resources that volume uses:
pxctl volume create --max_bandwidth <ReadBW>,<WriteBW> <volname/volid>
pxctl volume create --max_iops <ReadIOPS>,<WriteIOPS> <volname/volid>
The minimum value for --max_bandwidth
is 1
, and the minimum value for --max_iops
is 256
.
Update volumes using pxctl
If you want to add Application I/O Control to an existing volume, or change its throttling values, you can do so using the --max_bandwidth
and --max_iops
flags with the pxctl volume update
commands:
pxctl volume update --max_bandwidth <ReadBW>,<WriteBW> <volname/volid>
pxctl volume update --max_iops <ReadIOPS>,<WriteIOPS> <volname/volid>
If you want to limit only read or write bandwidth/IOPS traffic, set the other value to off
.
For example: Set read throttling for an existing volume to 1024 IOPS:
pxctl volume update --max_iops 1024,off exampleVolume
Remove throttling
If you want to remove throttling from a volume, set the value of the --max_bandwidth
and --max_iops
flags to off
for any throttling you wish to remove:
pxctl volume update --max_bandwidth off,<WriteBW> <volname/volid>
pxctl volume update --max_iops off,off <volname/volid>