Reference for autopilot in airgapped EKS
Summary and Key concepts
Summary:
This article describes the AutopilotRule Custom Resource Definition (CRD) used to automate storage management in Kubernetes environments with Portworx. It explains the key components of an AutopilotRule, including how to select the target objects and namespaces, define conditions that trigger actions, and specify the actions to take (such as resizing volumes or expanding storage pools). The article also details supported actions like resizing PVCs and expanding storage pools, with examples. Additionally, it outlines how to view and monitor Autopilot events to track rule activations and actions taken in real-time.
Kubernetes Concepts:
- Custom Resource Definition (CRD): Defines custom objects or resources that extend Kubernetes functionality, such as the
AutopilotRulefor Portworx. - PersistentVolumeClaim (PVC): A request for storage resources by a user in Kubernetes.
- Selector: A mechanism to select Kubernetes objects based on matching labels.
Portworx Concepts:
-
Autopilot: A Portworx feature that automates storage management tasks such as volume resizing or pool expansion based on pre-defined rules.
-
Volume Resize: An Autopilot action that increases the size of a PVC based on a condition, such as space utilization.
-
Storage Pool Expansion: An Autopilot action to expand the capacity of a Portworx storage pool by adding drives or resizing existing ones.
AutopilotRule CRD specification
| Field | Description | Optional? | Default |
|---|---|---|---|
| selector | Selects the objects affected by this rule using either matchLabels, matchExpressions, or both selectors. Syntax. | Yes | empty |
| namespaceSelector | Selects the namespaces affected by this rule using either matchLabels, matchExpressions, or both namespace selectors. Syntax. | Yes | all |
| conditions | Defines the metrics that need to be for the rule's actions to trigger. All conditions are AND'ed. Syntax. | No | |
| actions | Defines what action to take when the conditions are met. Syntax. See Supported Autopilot actions for all actions that you can specify here. | No | |
| pollInterval | Defines the interval in seconds at which the conditions for the rule are queried from the metrics provider. Note: You can reduce the poll interval to speed up Autopilot operations, but this will increase the load on Prometheus and lead to an increase in CPU consumption. | Yes |
|
| actionsCoolDownPeriod | Defines the duration in seconds for which autopilot will not re-trigger any actions once they have been executed. | Yes | 5 minutes |
selector
Selects the objects affected by this rule using either matchLabels, matchExpressions, or both selectors.
You can refer to Labels and Selectors for more information on how to use matchLabels and matchExpressions.
matchLabels
selector:
matchLabels:
<selector-key>: <selector-value>
matchExpressions
selector:
matchExpressions:
- key: <selector-key>
operator: <logical-operator>
values:
- <selector-value>
Example
Below is an example spec snippet for using matchLabels:
selector:
matchLabels:
app: postgres
Below is an example spec snippet for using matchExpressions:
selector:
matchExpressions:
- key: app
operator: In
values:
- data-writer
- writer
namespaceSelector
Selects the namespaces affected by this rule using a either matchLabels, matchExpressions, or both selectors.
You can refer to Labels and Selectors for more information on how to use matchLabels and matchExpressions.
matchLabels
namespaceSelector:
matchLabels:
<selector-key>: <selector-value>
matchExpressions
namespaceSelector:
matchExpressions:
- key: <selector-key>
operator: <logical-operator>
values:
- <selector-value>
Example
Below is an example spec snippet for using matchLabels:
namespaceSelector:
matchLabels:
app: postgres
Below is an example spec snippet for using matchExpressions:
namespaceSelector:
matchExpressions:
- key: app
operator: In
values:
- data-writer
- writer
conditions
Defines the metrics that need to be for the rule's actions to trigger.
Conditions compare the key field with the values field using the operator field. Condition keys can contain logic and use monitoring values.
conditions:
- key: "<condition-formula>"
operator: <logical-operator>
values:
- "<comparator>"
It follows the below schema.
| Field | Description | Optional? | Default |
|---|---|---|---|
| key | This is the metrics query that would be sent to the monitoring provider (e.g prometheus). | no | empty |
| operator | This is the logical operator to use to compare the results of the query in key above to the values. Supported operators are:
| no | empty |
| values | This is the value or list of values against which the key and operator are compared.
| no | empty |
Multiple conditions are combined using a logical AND.
actions
Defines what action to take when the conditions are met. See the Supported Actions section for the list of actions that you can specify.
action:
name: <operation>
params:
<operation-specific-paramater>: <value>
maxsize: "<value>Gi"
Supported Autopilot actions
openstorage.io.action.volume/resize
This action is to perform resize on Kubernetes PersistentVolumeClaims (PVCs).
Parameters
- scalepercentage: Specifies the percentage of current PVC size by which Autopilot should resize the PVC. If not specified, the default is 50%.
- maxsize: Specifies the maximum PVC size in bytes after which Autopilot should stop resizing the PVCs. Note that you can specify the unit of measurement as part of the value. For example, if you want to use GiB, you can specify the unit of measurement like this:
maxsize: "400Gi". If not specified, the default value is unlimited.
Examples
Resize the PVC by 100% of current size
actions:
- name: openstorage.io.action.volume/resize
params:
scalepercentage: "100"
maxsize: "12Gi"
openstorage.io.action.storagepool/expand
This action is to perform expansion on Portworx Storage Pools.
Parameters
- scalepercentage: Specifies the percentage of current Pool size by which Autopilot should resize it. If not specified, the default is 50%.
- scaletype: Specifies the type of operation to be performed to expand the pool. Supported values are:
- auto: Portworx scales the pool automatically by either adding new disks or resizing the existing disks. Autopilot chooses the best method to expand the pool based on the current configuration and available resources.
- add-drive: Portworx adds new disks to the existing storage pool.
- resize-drive: Portworx resizes the existing disks in the storage pool.
- scalesize: Specifies the amount, in Gi or Ti, by which Autopilot should expand a storage pool.
You cannot combine the scalepercentage and scalesize parameters; use only one of them in an Autopilot rule.
Examples
Expand the pool by 50% of current size automatically
actions:
- name: openstorage.io.action.storagepool/expand
params:
scalepercentage: "50"
scaletype: "auto"
Expand the pool by 50% of current size by adding disks
actions:
- name: openstorage.io.action.storagepool/expand
params:
scalepercentage: "50"
scaletype: "add-drive"
Expand the pool by 100Gi by resizing disks
actions:
- name: openstorage.io.action.storagepool/expand
params:
scalesize: "100Gi"
scaletype: "resize-drive"
Use cases
openstorage.io.action.storagepool/rebalance
This action performs a rebalance operation on Portworx Storage Pools.
Use cases
Autopilot Events
You can view the actions Autopilot takes by querying Autopilot events. These events provide insight into how your Autopilot rules are functioning, what actions they may be taking, and what actions they have taken in the past.
| Autopilot rule event | Description |
|---|---|
| Initializing | The rule's initial startup state where monitoring has not yet begun. |
| Normal | Autopilot is monitoring the rule as expected. |
| Triggered | The rule has its activation conditions met. |
| ActiveActionsPending | The rule's activation conditions have been met, but the actions are not yet being performed. |
| ActiveActionsTaken | Autopilot has performed the rule's actions, but still hasn't moved out of the active status. |
| ActionsDeclined | Autopilot has intentionally declined to perform a rule's action, for example when a PVC reaches a maximum user-defined size. |
| ActiveActionsInProgress | The rule is active and had its conditions met and there is an ongoing action on the object. |
| ActionNotLicensed | The action Autopilot is trying to perform is not permitted due to license restrictions |
You can query events from Kubernetes by entering the following kubectl get events command:
kubectl get events --field-selector involvedObject.kind=AutopilotRule
LAST SEEN FIRST SEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
41m 41m 1 pvc-total-size-15gi.15f13fcf9664716d AutopilotRule Normal Transition autopilot rule: pvc-total-size-15gi:pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29fda8e7 transition from Initializing => Normal
41m 41m 1 pvc-total-size-15gi.15f13fcf96a75e4f AutopilotRule Normal Transition autopilot rule: pvc-total-size-15gi:pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29fda8e7 transition from Initializing => Normal
36m 38m 2 pvc-total-size-15gi.15f14003ff20f5ec AutopilotRule Normal Transition autopilot rule: pvc-total-size-15gi:pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29fda8e7 transition from ActiveActionsInProgress => ActiveActionsTaken
35m 37m 2 pvc-total-size-15gi.15f140126cc4021c AutopilotRule Normal Transition autopilot rule: pvc-total-size-15gi:pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29fda8e7 transition from ActiveActionsTaken => Normal
35m 38m 3 pvc-total-size-15gi.15f13ff9ae4cc963 AutopilotRule Normal Transition autopilot rule: pvc-total-size-15gi:pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29fda8e7 transition from Normal => Triggered
34m 34m 2 pvc-total-size-15gi.15f14032de7660a7 AutopilotRule Normal Transition autopilot rule: pvc-total-size-15gi:pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29fda8e7 transition from ActiveActionsInProgress => ActionsDeclined