Skip to main content
Version: 3.1

Operating and Troubleshooting Autopilot

This section provides common operational procedures for monitoring and troubleshooting your autopilot installation.

Troubleshooting objects monitored by Autopilot

Follow the steps in the sections below to troubleshoot objects monitored by Autopilot

Get recent statuses using AutopilotRuleObjects

For each object monitored by Autopilot, it will create a corresponding autopilotruleobject instance in the namespace of the object.

  • For volumes (PVCs), the autopilotruleobject instance will be in the namespace of the PVC.
  • For storage pools, the autopilotruleobject instance will be in the namespace where Portworx is installed.

The autopilotruleobject is created only if an object's condiitons were atleast triggered once. In other words, you will not see an autopilotruleobject if the object was always in nornal state.

List all autopilotruleobjects

The following command lists all Autopilot rule objects in all namespaces:

kubectl get autopilotruleobjects --all-namespaces

Instead of entering the full autopilotruleobjects string, you can use the aro alias.

kubectl get aro --all-namespaces
NAMESPACE   NAME                                       AGE
pg1 pvc-ec475444-e2d6-4533-863b-03263da7b04c 1s

Describe a specific object

The Status section contains a list of recent object statuses:

kubectl describe aro -n pg1 pvc-ec475444-e2d6-4533-863b-03263da7b04c
Name:         pvc-ec475444-e2d6-4533-863b-03263da7b04c
Namespace: pg1
Labels: rule=volume-resize
Annotations: <none>
API Version:
Kind: AutopilotRuleObject
Creation Timestamp: 2020-08-26T22:29:45Z
Generation: 2
Owner References:
API Version:
Block Owner Deletion: true
Controller: true
Kind: AutopilotRule
Name: volume-resize
UID: 070e9932-5f6f-448c-a14a-62fbd2d5dbbc
Resource Version: 7554069
Self Link: /apis/
UID: 06030869-e1dd-4016-b12c-37fe57310baf
Last Process Timestamp: 2020-08-26T22:29:45Z
Message: rule: volume-resize:pvc-ec475444-e2d6-4533-863b-03263da7b04c transition from Normal => Triggered
State: Triggered
Last Process Timestamp: 2020-08-26T22:30:19Z
Message: rule: volume-resize:pvc-ec475444-e2d6-4533-863b-03263da7b04c transition from Triggered => ActionAwaitingApproval
State: ActionAwaitingApproval
Events: <none>

List autopilotruleobjects for a given autopilotrule

You can use the label selector rule=<RULE_NAME> for list autopilotruleobjects only for that autopilotrule.

kubectl get aro --all-namespaces -l rule=volume-resize
NAMESPACE   NAME                                       AGE
pg1 pvc-ec475444-e2d6-4533-863b-03263da7b04c 4m3s

Troubleshooting autopilot

  1. Create a directory (ap-cores) in which to store your support bundle files and send the support signal to the autopilot process:

    mkdir ap-cores
    POD=$(kubectl get pods -n kube-system -l name=autopilot | grep -v NAME | awk '{print $1}')
    kubectl exec -n kube-system $POD -- killall -SIGUSR1 autopilot
  2. Copy the support bundle files from your Kubernetes cluster to your directory:

    kubectl cp  kube-system/$POD:/tmp/ ap-cores/
    ls ap-cores
  3. Collect and place your autopilot pod logs into an autopilot-pod.log file within your temporary directory:

    kubectl logs $POD -n kube-system --tail=99999 > ap-cores/autopilot-pod.log

Once you've created a support bundle and collected your logs, send all of the files in the ap-cores/ directory to Portworx support.

Was this page helpful?