Autopilot action approvals using kubectl in AWS EKS
Prerequisites
- Autopilot 1.3.0 and above
Overview
The general workflow of using an AutopilotRule with approvals enabled consists of the following:
- Create AutopilotRule with approvals enabled
- Approve or Decline the action by using the ActionApproval CRD
The general workflow expands to the following steps. The Example section later will cover a detailed working example.
- Create an AutopilotRule with
enforcement: approvalRequired
in the spec - Wait until the objects meet the conditions specified in the rule. For example, if the rule is to expand a volume when its usage is greater than 50%, wait for this condition.
- Once the conditions are met, list of the action approvals in the namespace. Identity the item in the list for the concerned object.
- Update the
approvalState
field in the ActionApproval object spec toapproved
ordeclined
. - Based on whether you approved or declined in the previous step, the action will either proceed or get declined respectively.
Example
The example below demonstrates an AutopilotRule that expands Postgres PVCs whose usage increases more than 50%. The rule will require approvals before any action to expand the PVC can take place.
Create specs
Application and PVC specs
Create the storage and application spec files:
-
Create
namespace.yaml
and place the following content inside it:apiVersion: v1
kind: Namespace
metadata:
name: pg1
labels:
type: db -
Create
postgres-sc.yaml
and place the following content inside it:##### Portworx storage class
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: postgres-pgbench-sc
provisioner: kubernetes.io/portworx-volume
parameters:
repl: "2"
allowVolumeExpansion: true -
Create
postgres-vol.yaml
and place the following content inside it:kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pgbench-data
labels:
app: postgres
spec:
storageClassName: postgres-pgbench-sc
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pgbench-state
spec:
storageClassName: postgres-pgbench-sc
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi -
Create
postgres-app.yaml
and place the following content inside it. Note the following:-
The application in this example is a PostgreSQL database with a pgbench sidecar.
-
The
SIZE
environment variable in this spec instructs pgbench to write 8GiB of data to the volume. Since the PVC is only 10GiB in size, Autopilot will resize the PVC when needed.apiVersion: apps/v1
kind: Deployment
metadata:
name: pgbench
labels:
app: pgbench
spec:
selector:
matchLabels:
app: pgbench
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
replicas: 1
template:
metadata:
labels:
app: pgbench
spec:
schedulerName: stork
containers:
- image: postgres:9.5
name: postgres
ports:
- containerPort: 5432
env:
- name: POSTGRES_USER
value: pgbench
- name: POSTGRES_PASSWORD
value: superpostgres
- name: PGBENCH_PASSWORD
value: superpostgres
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: pgbenchdb
- name: pgbench
image: portworx/torpedo-pgbench:latest
imagePullPolicy: "Always"
env:
- name: PG_HOST
value: 127.0.0.1
- name: PG_USER
value: pgbench
- name: SIZE
value: "8"
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: pgbenchdb
- mountPath: /pgbench
name: pgbenchstate
volumes:
- name: pgbenchdb
persistentVolumeClaim:
claimName: pgbench-data
- name: pgbenchstate
persistentVolumeClaim:
claimName: pgbench-state
-
AutopilotRule spec
Once you've created your storage and application specs, you can create an AutopilotRule that controls them.
Create a YAML spec for the autopilot rule named autopilotrule-approval-example.yaml
and place the following content inside it:
apiVersion: autopilot.libopenstorage.org/v1alpha1
kind: AutopilotRule
metadata:
name: volume-resize
spec:
#### enforcement indicates that actions from this rule need approval
enforcement: approvalRequired
##### selector filters the objects affected by this rule given labels
selector:
matchLabels:
app: postgres
##### namespaceSelector selects the namespaces of the objects affected by this rule
namespaceSelector:
matchLabels:
type: db
##### conditions are the symptoms to evaluate. All conditions are AND'ed
conditions:
# volume usage should be less than 50%
expressions:
- key: "100 * (px_volume_usage_bytes / px_volume_capacity_bytes)"
operator: Gt
values:
- "50"
##### action to perform when condition is true
actions:
- name: openstorage.io.action.volume/resize
params:
# resize volume by scalepercentage of current size
scalepercentage: "100"
# volume capacity should not exceed 400GiB
maxsize: "400Gi"
Apply specs
Once you've designed your specs, deploy them:
kubectl apply -f autopilotrule-approval-example.yaml
kubectl apply -f namespace.yaml
kubectl apply -f postgres-sc.yaml
kubectl apply -f postgres-vol.yaml -n pg1
kubectl apply -f postgres-app.yaml -n pg1