Reference for autopilot in airgapped EKS

Summary and Key concepts

Summary:

This article describes the AutopilotRule Custom Resource Definition (CRD) used to automate storage management in Kubernetes environments with Portworx. It explains the key components of an AutopilotRule, including how to select the target objects and namespaces, define conditions that trigger actions, and specify the actions to take (such as resizing volumes or expanding storage pools). The article also details supported actions like resizing PVCs and expanding storage pools, with examples. Additionally, it outlines how to view and monitor Autopilot events to track rule activations and actions taken in real-time.

Kubernetes Concepts:

• Custom Resource Definition (CRD): Defines custom objects or resources that extend Kubernetes functionality, such as the AutopilotRule for Portworx.
• PersistentVolumeClaim (PVC): A request for storage resources by a user in Kubernetes.
• Selector: A mechanism to select Kubernetes objects based on matching labels.

Portworx Concepts:

• Autopilot: A Portworx feature that automates storage management tasks such as volume resizing or pool expansion based on pre-defined rules.

• Volume Resize: An Autopilot action that increases the size of a PVC based on a condition, such as space utilization.

• Storage Pool Expansion: An Autopilot action to expand the capacity of a Portworx storage pool by adding drives or resizing existing ones.

AutopilotRule CRD specification

Field	Description	Optional?	Default
selector	Selects the objects affected by this rule using a matchLabels label selector. Syntax.	yes	empty
namespaceSelector	Selects the namespaces affected by this rule using a matchLabels label selector. Syntax.	yes	all
conditions	Defines the metrics that need to be for the rule's actions to trigger. All conditions are AND'ed. Syntax.	no
actions	Defines what action to take when the conditions are met. Syntax. See Supported Autopilot actions for all actions that you can specify here.	no
pollInterval	Defines the interval in seconds at which the conditions for the rule are queried from the metrics provider.	yes	10 seconds
actionsCoolDownPeriod	Defines the duration in seconds for which autopilot will not re-trigger any actions once they have been executed.	yes	5 minutes

selector

Selects the objects affected by this rule using a matchLabels label selector.

selector:
  matchLabels:
    <selector-key>: <selector-value>

namespaceSelector

Selects the namespaces affected by this rule using a matchLabels label selector.

namespaceSelector:
  matchLabels:
    <selector-key>:<selector-value>

conditions

Defines the metrics that need to be for the rule's actions to trigger.

Conditions compare the key field with the values field using the operator field. Condition keys can contain logic and use monitoring values.

conditions:
  - key: "<condition-formula>"
    operator: <logical-operator>
    values:
    - "<comparator>"

It follows the below schema.

Field	Description	Optional?	Default
key	This is the metrics query that would be sent to the monitoring provider (e.g prometheus).	no	empty
operator	This is the logical operator to use to compare the results of the query in key above to the values. Supported operators are: In NotIn Lt Gt LtEq GtEq InRange NotInRange	no	empty
values	This is the value or list of values against which the key and operator are compared. In, NotIn need a list of values Gt, Lt, GtEq, LtEq need a single numerical value. InRange, NotInRange need exactly 2 numerical values	no	empty

note

Multiple conditions are combined using a logical AND.

actions

Defines what action to take when the conditions are met. See the Supported Actions section for the list of actions that you can specify.

action:
  name: <operation>
  params:
    <operation-specific-paramater>: <value>
    maxsize: "<value>Gi"

Supported Autopilot actions

openstorage.io.action.volume/resize

This action is to perform resize on Kubernetes PersistentVolumeClaims (PVCs).

Parameters

• scalepercentage: Specifies the percentage of current PVC size by which Autopilot should resize the PVC. If not specified, the default is 50%.
• maxsize: Specifies the maximum PVC size in bytes after which Autopilot should stop resizing the PVCs. Note that you can specify the unit of measurement as part of the value. For example, if you want to use GiB, you can specify the unit of measurement like this: maxsize: "400Gi". If not specified, the default value is unlimited.

Examples

Resize the PVC by 100% of current size

  actions:
  - name: openstorage.io.action.volume/resize
    params:
      scalepercentage: "100"
      maxsize: "12Gi"

openstorage.io.action.storagepool/expand

This action is to perform expansion on Portworx Storage Pools.

Parameters

• scalepercentage: Specifies the percentage of current Pool size by which Autopilot should resize it. If not specified, the default is 50%.
• scaletype: Specifies the type of operation to be performed to expand the pool. Supported values are:
  • auto: Portworx scales the pool automatically by either adding new disks or resizing the existing disks. Autopilot chooses the best method to expand the pool based on the current configuration and available resources.
  • add-drive: Portworx adds new disks to the existing storage pool.
  • resize-drive: Portworx resizes the existing disks in the storage pool.
• scalesize: Specifies the amount, in Gi or Ti, by which Autopilot should expand a storage pool.

note

You cannot combine the scalepercentage and scalesize parameters; use only one of them in an Autopilot rule.

Examples

Expand the pool by 50% of current size automatically

  actions:
  - name: openstorage.io.action.storagepool/expand
    params:
      scalepercentage: "50"
      scaletype: "auto"

Expand the pool by 50% of current size by adding disks

  actions:
  - name: openstorage.io.action.storagepool/expand
    params:
      scalepercentage: "50"
      scaletype: "resize-drive"

Expand the pool by 100Gi by resizing disks

  actions:
  - name: openstorage.io.action.storagepool/expand
    params:
      scalesize: "100Gi"
      scaletype: "resize-drive"

Use cases

• Automatically expand storage pools by usage
• Expand all storage pools

openstorage.io.action.storagepool/rebalance

This action performs a rebalance operation on Portworx Storage Pools.

Use cases

• Automatically rebalance storage pools based on provisioned and used space

Autopilot Events

You can view the actions Autopilot takes by querying Autopilot events. These events provide insight into how your Autopilot rules are functioning, what actions they may be taking, and what actions they have taken in the past.

Autopilot rule event	Description
Initializing	The rule's initial startup state where monitoring has not yet begun.
Normal	Autopilot is monitoring the rule as expected.
Triggered	The rule has its activation conditions met.
ActiveActionsPending	The rule's activation conditions have been met, but the actions are not yet being performed.
ActiveActionsTaken	Autopilot has performed the rule's actions, but still hasn't moved out of the active status.
ActionsDeclined	Autopilot has intentionally declined to perform a rule's action, for example when a PVC reaches a maximum user-defined size.
ActiveActionsInProgress	The rule is active and had its conditions met and there is an ongoing action on the object.
ActionNotLicensed	The action Autopilot is trying to perform is not permitted due to license restrictions

You can query events from Kubernetes by entering the following kubectl get events command:

kubectl get events --field-selector involvedObject.kind=AutopilotRule

LAST SEEN   FIRST SEEN   COUNT   NAME                                   KIND            SUBOBJECT   TYPE      REASON           SOURCE      MESSAGE
41m         41m          1       pvc-total-size-15gi.15f13fcf9664716d   AutopilotRule               Normal    Transition       autopilot   rule: pvc-total-size-15gi:pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29fda8e7 transition from Initializing => Normal
41m         41m          1       pvc-total-size-15gi.15f13fcf96a75e4f   AutopilotRule               Normal    Transition       autopilot   rule: pvc-total-size-15gi:pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29fda8e7 transition from Initializing => Normal
36m         38m          2       pvc-total-size-15gi.15f14003ff20f5ec   AutopilotRule               Normal    Transition       autopilot   rule: pvc-total-size-15gi:pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29fda8e7 transition from ActiveActionsInProgress => ActiveActionsTaken
35m         37m          2       pvc-total-size-15gi.15f140126cc4021c   AutopilotRule               Normal    Transition       autopilot   rule: pvc-total-size-15gi:pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29fda8e7 transition from ActiveActionsTaken => Normal
35m         38m          3       pvc-total-size-15gi.15f13ff9ae4cc963   AutopilotRule               Normal    Transition       autopilot   rule: pvc-total-size-15gi:pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29fda8e7 transition from Normal => Triggered
34m         34m          2       pvc-total-size-15gi.15f14032de7660a7   AutopilotRule               Normal    Transition       autopilot   rule: pvc-total-size-15gi:pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29fda8e7 transition from ActiveActionsInProgress => ActionsDeclined