Version: 3.2

Autopilot action approvals using kubectl in AWS EKS

Summary and Key concepts

Summary:

This article explains how to use Portworx Autopilot rules with approval workflows. It covers creating an AutopilotRule that requires manual approval before taking actions, such as expanding a PVC. The workflow includes creating the application, storage class, PVCs, and AutopilotRule, then monitoring the rule as it triggers when specified conditions (like volume usage exceeding 50%) are met. Afterward, the article demonstrates how to approve or decline the action via the ActionApproval resource. Once approved, the action (e.g., volume resizing) proceeds; otherwise, it remains in a declined state until the ActionApproval object is deleted.

Kubernetes Concepts:

PersistentVolumeClaim (PVC): A request for storage resources in Kubernetes. Autopilot can resize PVCs based on predefined rules.
StorageClass: Defines different types of storage that can be dynamically provisioned in Kubernetes.
kubectl patch: A command used to modify Kubernetes resources, such as approving or declining an ActionApproval.

Portworx Concepts:

Autopilot: Automates storage operations like resizing PVCs based on rules and conditions.

Overview

The general workflow of using an AutopilotRule with approvals enabled consists of the following:

Create AutopilotRule with approvals enabled
Approve or Decline the action by using the ActionApproval CRD

The general workflow expands to the following steps. The Example section later will cover a detailed working example.

Create an AutopilotRule with enforcement: approvalRequired in the spec
Wait until the objects meet the conditions specified in the rule. For example, if the rule is to expand a volume when its usage is greater than 50%, wait for this condition.
Once the conditions are met, list of the action approvals in the namespace. Identity the item in the list for the concerned object.
Update the approvalState field in the ActionApproval object spec to approved or declined.
Based on whether you approved or declined in the previous step, the action will either proceed or get declined respectively.

Example

The example below demonstrates an AutopilotRule that expands Postgres PVCs whose usage increases more than 50%. The rule will require approvals before any action to expand the PVC can take place.

Create specs

Application and PVC specs

Create the storage and application spec files:

Create namespace.yaml and place the following content inside it:

apiVersion: v1
kind: Namespace
metadata:
  name: pg1
  labels:
    type: db

Create postgres-sc.yaml and place the following content inside it:

##### Portworx storage class
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: postgres-pgbench-sc
provisioner: pxd.portworx.com
parameters:
  repl: "2"
allowVolumeExpansion: true

Create postgres-vol.yaml and place the following content inside it:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pgbench-data
  labels:
    app: postgres
spec:
  storageClassName: postgres-pgbench-sc
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pgbench-state
spec:
  storageClassName: postgres-pgbench-sc
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Createpostgres-app.yaml and place the following content inside it. Note the following:

The application in this example is a PostgreSQL database with a pgbench sidecar.

The SIZE environment variable in this spec instructs pgbench to write 8GiB of data to the volume. Since the PVC is only 10GiB in size, Autopilot will resize the PVC when needed.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pgbench
  labels:
    app: pgbench
spec:
  selector:
    matchLabels:
      app: pgbench
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  replicas: 1
  template:
    metadata:
      labels:
        app: pgbench
    spec:
      schedulerName: stork
      containers:
        - image: postgres:9.5
          name: postgres
          ports:
          - containerPort: 5432
          env:
          - name: POSTGRES_USER
            value: pgbench
          - name: POSTGRES_PASSWORD
            value: superpostgres
          - name: PGBENCH_PASSWORD
            value: superpostgres
          - name: PGDATA
            value: /var/lib/postgresql/data/pgdata
          volumeMounts:
          - mountPath: /var/lib/postgresql/data
            name: pgbenchdb
        - name: pgbench
          image: portworx/torpedo-pgbench:latest
          imagePullPolicy: "Always"
          env:
            - name: PG_HOST
              value: 127.0.0.1
            - name: PG_USER
              value: pgbench
            - name: SIZE
              value: "8"
          volumeMounts:
          - mountPath: /var/lib/postgresql/data
            name: pgbenchdb
          - mountPath: /pgbench
            name: pgbenchstate
      volumes:
      - name: pgbenchdb
        persistentVolumeClaim:
          claimName: pgbench-data
      - name: pgbenchstate
        persistentVolumeClaim:
          claimName: pgbench-state

AutopilotRule spec

Once you've created your storage and application specs, you can create an AutopilotRule that controls them.

Create a YAML spec for the autopilot rule named autopilotrule-approval-example.yaml and place the following content inside it:

apiVersion: autopilot.libopenstorage.org/v1alpha1
kind: AutopilotRule
metadata:
  name: volume-resize
spec:
  #### enforcement indicates that actions from this rule need approval
  enforcement: approvalRequired
  ##### selector filters the objects affected by this rule given labels
  selector:
    matchLabels:
      app: postgres
  ##### namespaceSelector selects the namespaces of the objects affected by this rule
  namespaceSelector:
    matchLabels:
      type: db
  ##### conditions are the symptoms to evaluate. All conditions are AND'ed
  conditions:
    # volume usage should be less than 50%
    expressions:
    - key: "100 * (px_volume_usage_bytes / px_volume_capacity_bytes)"
      operator: Gt
      values:
        - "50"
  ##### action to perform when condition is true
  actions:
  - name: openstorage.io.action.volume/resize
    params:
      # resize volume by scalepercentage of current size
      scalepercentage: "100"
      # volume capacity should not exceed 400GiB
      maxsize: "400Gi"

Apply specs

Once you've designed your specs, deploy them:

kubectl apply -f autopilotrule-approval-example.yaml
kubectl apply -f namespace.yaml
kubectl apply -f postgres-sc.yaml
kubectl apply -f postgres-vol.yaml -n pg1
kubectl apply -f postgres-app.yaml -n pg1

Approve or decline the action

Wait until conditions are triggered

After you apply the specs above, Postgres will start populating data to the PVC. Once Autopilot detects that the volume usage is greater than 50%, it will create an ActionApproval object in the pg1 namespace.

List the Kubernetes events for this rule and wait until your rule is in the ActionAwaitingApproval state.

kubectl get events --field-selector involvedObject.kind=AutopilotRule,involvedObject.name=volume-resize -n default -w

LAST SEEN   TYPE     REASON       OBJECT                        MESSAGE
10m         Normal   Transition   autopilotrule/volume-resize   rule: volume-resize:pvc-xxxxxxxx-xxxx-xxxx-xxxx-9ddd748cfe8e transition from Initializing => Normal
67s         Normal   Transition   autopilotrule/volume-resize   rule: volume-resize:pvc-xxxxxxxx-xxxx-xxxx-xxxx-9ddd748cfe8e transition from Normal => Triggered
34s         Normal   Transition   autopilotrule/volume-resize   rule: volume-resize:pvc-xxxxxxxx-xxxx-xxxx-xxxx-9ddd748cfe8e transition from Triggered => ActionAwaitingApproval

If you only see Initializing => Normal as the event, this means postgres is still writing data to your volume and usage has not crossed 50%.

Approve the action

List the actionapproval for this object:

kubectl get actionapproval -n pg1

NAME                                                     APPROVAL-STATE
volume-resize-pvc-xxxxxxxx-xxxx-xxxx-xxxx-9ddd748cfe8e   pending

Patch and approve the actionapproval:

kubectl patch actionapproval -n pg1 volume-resize-pvc-xxxxxxxx-xxxx-xxxx-xxxx-9ddd748cfe8e --type=merge -p '{"spec":{"approvalState":"approved"}}' 

actionapproval.autopilot.libopenstorage.org/volume-resize-pvc-xxxxxxxx-xxxx-xxxx-xxxx-9ddd748cfe8e patched

Once approved, you will see that the actions will progress. List the events again:

kubectl get events --field-selector involvedObject.kind=AutopilotRule,involvedObject.name=volume-resize -n default -w

LAST SEEN   TYPE     REASON       OBJECT                        MESSAGE
19m         Normal   Transition   autopilotrule/volume-resize   rule: volume-resize:pvc-xxxxxxxx-xxxx-xxxx-xxxx-9ddd748cfe8e transition from Initializing => Normal
10m         Normal   Transition   autopilotrule/volume-resize   rule: volume-resize:pvc-xxxxxxxx-xxxx-xxxx-xxxx-9ddd748cfe8e transition from Normal => Triggered
9m47s       Normal   Transition   autopilotrule/volume-resize   rule: volume-resize:pvc-xxxxxxxx-xxxx-xxxx-xxxx-9ddd748cfe8e transition from Triggered => ActionAwaitingApproval
8m52s       Normal   Transition   autopilotrule/volume-resize   rule: volume-resize:pvc-xxxxxxxx-xxxx-xxxx-xxxx-9ddd748cfe8e transition from ActionAwaitingApproval => ActiveActionsPending
7m51s       Normal   Transition   autopilotrule/volume-resize   rule: volume-resize:pvc-xxxxxxxx-xxxx-xxxx-xxxx-9ddd748cfe8e transition from ActiveActionsPending => ActiveActionsInProgress
7m20s       Normal   Transition   autopilotrule/volume-resize   rule: volume-resize:pvc-xxxxxxxx-xxxx-xxxx-xxxx-9ddd748cfe8e transition from ActiveActionsInProgress => ActiveActionsTaken

Decline the action

To decline, you should use declined instead of approved in the patch command.

Actions for the object will continue to stay in declined state until the actionapproval object is present and has approval state as declined. When you want Autopilot to resume monitoring this object, delete the actionapproval object.

For e.g for the above case,

kubectl delete actionapproval -n pg1 volume-resize-pvc-xxxxxxxx-xxxx-xxxx-xxxx-9ddd748cfe8e

Summary:​

Kubernetes Concepts:​

Portworx Concepts:​

Overview​

Example​

Create specs​

Application and PVC specs​

AutopilotRule spec​

Apply specs​

Approve or decline the action​

Wait until conditions are triggered​

Approve the action​

Decline the action​