Portworx Autopilot Release Notes

1.3.16

December 9, 2024

Improvements

Issue Number	Issue Description
PWX-38984	Added support for `matchExpressions` along with `matchLabels` in Autopilot rules. You can now select the objects using either `matchLabels`, `matchExpressions`, or both selectors.
PWX-34122	Updated the CRD schema to adhere to the strict structural schema validations introduced in Kubernetes 1.23 and later. This allows setting `actionCoolDownPeriod` and `pollInterval` without encountering strict decoding errors.
PWX-38947	Autopilot now supports `add-drive` and `resize-drive` options for pool expansion. Additionally, the default resize type is changed to `auto`.
PWX-35367	Implemented a dedicated scheduler process for moving FSM states, which would decrease the latency of Autopilot operations.
PWX-39812	The following changes has been made to reduce the CPU usage of the Prometheus pod: Increased the default value of `pollInterval` from 10 seconds to 60 seconds to reduce the CPU usage of the Prometheus pod when scraping a lot of data due to a large number of volumes. Optimization to reduce the number of query calls from Autopilot to Prometheus.

Fixes

Issue Number	Issue Description	Severity
PWX-39803	If the rebalance action is skipped due to an error during the dry run without checking the audit actions, the cluster might be left in an unbalanced state. User Impact: Impacted overall performance and resource distribution across the cluster. Resolution: Added checks to ensure that rebalance is not skipped when audit entries are present in the dry run response. Affected Versions: All	Minor
PWX-36918	Autopilot assumed the default `.svc.cluster.local` DNS domain for the Kubernetes cluster, causing connection failures using different DNS domains. User Impact: Connection failures were seen when using a DNS domain other than the default domain. Resolution: Autopilot now determines the DNS domain Kubernetes uses for its internal services. Affected Versions: All	Major
PWX-35742	Autopilot triggered multiple dry run rebalance operations, even if there is a run in progress. User Impact: Multiple dry run rebalance operations were trigerred continuously. Resolution: Added checks to ensure that no other rebalance operation is running before triggering a new one. Affected Versions: All	Minor

1.3.15

July 25, 2024

Notes

In order to use Portworx with OpenShift version 4.16, you must use Operator version 24.1.1 and Autopilot version 1.3.15.

1.3.14

March 18, 2024

Improvements

Improvement Number	Improvement Description
PWX-36274	OpenShift users on version 4.12 or newer, can now configure Autopilot to use OpenShift Prometheus deployment for monitoring.

Fixes

The following issues have been fixed:

Issue Number Issue Description

PWX-36160 In certain scenarios, Autopilot would generate duplicate AutopilotRuleObjects for Storage pools. The pool expansion and re-balance rules would overlap, generating events on a common AutopilotRuleObjects. This overlap prevented any of the rules from proceeding and taking action.

Resolution: Autopilot now does not create duplicate AutopilotRuleObjects for storage pools.

PWX-36046 Prometheus self-signed CA certificate bundle was inadvertently overriding the container's default CA certificates. As a result, while Autopilot could integrate with SSL-enabled Prometheus instances using self-signed SSL certificates, it would fail to work with those using commercially signed SSL certificates.

Resolution: Autopilot is now compatible with both commercial and self-signed SSL-enabled Prometheus endpoints.

Issue Number	Issue Description
PWX-36160	In certain scenarios, Autopilot would generate duplicate `AutopilotRuleObjects` for Storage pools. The pool expansion and re-balance rules would overlap, generating events on a common `AutopilotRuleObjects`. This overlap prevented any of the rules from proceeding and taking action. Resolution: Autopilot now does not create duplicate `AutopilotRuleObjects` for storage pools.
PWX-36046	Prometheus self-signed CA certificate bundle was inadvertently overriding the container's default CA certificates. As a result, while Autopilot could integrate with SSL-enabled Prometheus instances using self-signed SSL certificates, it would fail to work with those using commercially signed SSL certificates. Resolution: Autopilot is now compatible with both commercial and self-signed SSL-enabled Prometheus endpoints.

Known issues (Errata)

Issue Number Issue Description

PD-2814 When attempting to resize a pool or volume using AutopilotRules, users may encounter errors leading to a failed condition check for a rule and see the connection reset by peer error message when in the Kubernetes events by running the command kubectl get events --field-selector involvedObject.kind=AutopilotRule --all-namespaces. Autopilot may enter a state where it does not retry the operation after encountering this error.

Workaround: Manually restart the Autopilot pod within your Kubernetes environment. This action forces Autopilot to reconnect to the OCP Prometheus endpoints.

Issue Number	Issue Description
PD-2814	When attempting to resize a pool or volume using AutopilotRules, users may encounter errors leading to a failed condition check for a rule and see the `connection reset by peer` error message when in the Kubernetes events by running the command `kubectl get events --field-selector involvedObject.kind=AutopilotRule --all-namespaces`. Autopilot may enter a state where it does not retry the operation after encountering this error. Workaround: Manually restart the Autopilot pod within your Kubernetes environment. This action forces Autopilot to reconnect to the OCP Prometheus endpoints.

1.3.13

February 08, 2024

Notes

In order to use Portworx with OpenShift version 4.14, you must use Operator version 23.10.3 and Autopilot version 1.3.13. The Upgrade to OpenShift 4.14 or newer with Portworx page details the steps for upgrading OpenShift.
Enabling monitoring on OpenShift version 4.14 will no longer setup a separate Prometheus instance for metrics and Autopilot.

1.3.12

November 30, 2023

Improvements

Improvement Number	Improvement Description
PWX-34605	Autopilot now appends the rule name as a suffix to newly created AutopilotRuleObjects. As a result, Autopilot can schedule and execute multiple rules for a pool without overriding them.
PWX-33620	The time required to resize PVCs in a cluster with a large number of volumes has been considerably reduced.

1.3.11

September 25, 2023

Improvements

Improvement Number	Improvement Description
PWX-33668	Autopilot now does not treat an offline Prometheus target as a fatal error. It continues to monitor metrics from other Prometheus targets.

Fixes

The following issues have been fixed:

Issue Number	Issue Description
PWX-33620	With Autopilot, resizing of PVCs took a long time in a cluster with a large number of PVCs. User impact: Autopilot's Kubernetes API calls were being throttled adding to the delay required for triggering actions. Resolution: Added controller-runtime caching to Autopilot to reduce the time taken for volume resizing.

1.3.10

August 16, 2023

Notes

This version addresses security vulnerabilities.

1.3.9

August 7, 2023

Notes

This version addresses security vulnerabilities.

1.3.8

May 31, 2023

Fixes

The following issues have been fixed:

Issue Number	Issue Description
PWX-31468	Autopilot pods kept restarting when deployed on OpenShift with the `fsGroup securityContext` setting. User impact: Autopilot pods crashed because it used a filesystem watcher to monitor configuration changes and load applications. Resolution: Autopilot now uses the `sched-ops` API to monitor configuration changes.

1.3.7

February 22, 2023

Notes

This version addresses security vulnerabilities.

1.3.6

February 16, 2023

Notes

This version addresses security vulnerabilities.

1.3.5

January 26, 2023

Improvements

Portworx has upgraded or enhanced functionality in the following areas:

Improvement Number	Improvement Description
PWX-28744	Miscellaneous improvements.

1.3.3

September 26, 2022

Fixes

The following issues have been fixed:

Issue Number	Issue Description
PWX-26135	Parameters in AutopilotRule actions were pruned for Kubernetes versions 1.23 and higher. User impact: Users on Kubernetes 1.23 or greater saw some parameters, such as 'x-kubernetes-preserve-unknown-fields: true', removed when applying a CRD. Resolution: Autopilot no longer prunes parameters.

1.3.2

December 14, 2021

Improvements

Portworx has upgraded or enhanced functionality in the following areas:

Improvement Number	Improvement Description
PD-1047	Autopilot 1.3.2 now supports installations that use PX-Security. Follow the instructions in Autopilot Install and Setup to use Autopilot with PX-Security.

1.3.1

November 22, 2021

Improvements

Portworx has upgraded or enhanced functionality in the following areas:

Improvement Number	Improvement Description
PD-1035	Autopilot 1.3.1 now supports Kubernetes 1.22 and higher

1.3.0

Sep 02, 2020

Improvements

Portworx has upgraded or enhanced functionality in the following areas:

Improvement Number	Improvement Description
AUT-113	Added support to rebalance Portworx storage pools. Refer to documentation for more details.
AUT-136	Added support to users to require approval before actions are taken for a AutopilotRule. Refer to documentation for more details.
AUT-138	Added support for `requiredMatches` in the `AutopilotRule` CRD. This allows users to specify number of conditions that are required to match in the rule. See the Automatically rebalance Portworx storage pools use case on example on how this can be used.
AUT-144	Added support for Autopilot to create Github Pull Requests to approvals for its actions. This enables users to integrate GitOps workflows with Autopilot. for more details.
AUT-157	Make AutopilotRuleObject namespace scoped. Refer to documentation for usage of AutopilotRuleObjects.
AUT-205	Improve support bundle collection

Fixes

The following issues have been fixed:

Issue Number	Issue Description
AUT-83	Autopilot pool expand should never bring Portworx out of quorum User impact: In certain situations where 2 or more pools have non-intersection volumes, Autopilot can triggered expand on multiple pools at the same time which can bring the PX cluster out of quorum. Resolution: Autopilot will now perform expansion on only one pool at a given time. Subsequent pools will have their actions in pending state until the previous one is complete.
AUT-87	If an action is declined, Autopilot will now perform exponential backoff before retrying it. User impact: If an action was declined due to the maxsize being hit in the AutopilotRule for a PVC resize, Autopilot used to aggressively retry flooding logs and events. Resolution: Autopilot will now perform exponential backoff.
AUT-169	Autopilot loses track of volume for rule when PX-Backup backs up and restores to same namespace. User impact: If a user has a PVC that was restored from a backup or pre-provisioned, Autopilot would not be able to track the metrics of the PVC correctly. Resolution: Autopilot was incorrect assuming the volume name of a PVC by using the PVC UUID. Instead now, it will use the actual volume name from the PVC spec.
AUT-209	Fixed an issue where Autopilot incorrectly determines that conditions are not met for an object. User impact: There are situations where the prometheus API reports empty metrics when it is initializing and it hasn't scrapped its targets. If Autopilot queries prometheus during such times, it will find empty metrics and will be misled to consider the conditions as not being met for that object. Resolution: Autopilot will no longer accept metrics from Prometheus if a Portworx metrics target is not healthy. The Portworx metrics target will become healthy when Prometheus has initialized completely and has been able to perform a successful scrape.

1.2.1

May 30, 2020

Improvements

Portworx has upgraded or enhanced functionality in the following areas:

Improvement Description
The PVC resize action no longer requires a Portworx Autopilot Capacity Management license

1.2.0

March 20, 2020

Improvements

Portworx has upgraded or enhanced functionality in the following areas:

Improvement Description
Added support for a new parameter, `scalesize`, under the storage pool expand action that allows you to increase storage pool capacity by a fixed amount. See details on all action parameters here.
Added validation for the AutopilotRule CRD
Added an alert event that occurs when Autopilot detects an action that cannot be performed due to license restrictions. See the list of alerts for more information.

1.1.0

February 19, 2020

Improvements

Portworx has upgraded or enhanced functionality in the following areas:

Improvement Description
Added a new CRD, called AutopilotRuleObject, which can be used to check for useful events in objects that autopilot monitors, such as PVCs and StoragePools.
Added basic metrics for monitoring Autopilot and Grafana dashboards to view them. To view Autopilot metrics, follow the steps in the Prometheus and Grafana article.

Fixes

The following issues have been fixed:

Issue Description
When an Autopilot pod restarted while expanding a storage pool, it sometimes started expanding another storage pool. User impact: If multiple pools for a volume started expansion together, the volume could go out of quorum. Resolution: Autopilot pods now correctly wait for previous storage pool expansions to complete when they're restarted.
Deleting Autopilot rules or PVCs sometimes caused the Autopilot pod to crash User impact: If Autopilot crashed, it could start expanding another storage pool while one was already in being expanded. Resolution: The Autopilot pod no longer crashes when rules or PVCs are deleted.

1.0.0

November 18, 2019

Introducing Portworx Autopilot! See the Autopilot section of the documentation for more information

1.3.16​

Improvements​

Fixes​

1.3.15​

Notes​

1.3.14​

Improvements​

Fixes​

Known issues (Errata)​

1.3.13​

Notes​

1.3.12​

Improvements​

1.3.11​

Improvements​

Fixes​

1.3.10​

Notes​

1.3.9​

Notes​

1.3.8​

Fixes​

1.3.7​

Notes​

1.3.6​

Notes​

1.3.5​

Improvements​

1.3.3​

Fixes​

1.3.2​

Improvements​

1.3.1​

Improvements​

1.3.0​

Improvements​

Fixes​

1.2.1​

Improvements​

1.2.0​

Improvements​

1.1.0​

Improvements​

Fixes​

1.0.0​

1.3.16

Improvements

Fixes

1.3.15

Notes

1.3.14

Improvements

Fixes

Known issues (Errata)

1.3.13

Notes

1.3.12

Improvements

1.3.11

Improvements

Fixes

1.3.10

Notes

1.3.9

Notes

1.3.8

Fixes

1.3.7

Notes

1.3.6

Notes

1.3.5

Improvements

1.3.3

Fixes

1.3.2

Improvements

1.3.1

Improvements

1.3.0

Improvements

Fixes

1.2.1

Improvements

1.2.0

Improvements

1.1.0

Improvements

Fixes

1.0.0