Portworx Operator Release Notes
23.10.1
November 22, 2023
Fixes
Portworx Operator has resolved the following issues in this release:
Issue Number | Issue Description |
---|---|
PWX-35067 | In certain scenarios, the Operator incorrectly auto-detected the cloud provider when using the cloud drive feature. This happened when users left the spec.cloudStorage.provider field blank and were running on any of the following environments:
User impact: If you encounter this issue, Portworx will fail to come up successfully during a fresh install or update. Resolution: Portworx no longer auto detects cloud providers. |
23.10.0
November 7, 2023
In certain scenarios, the Operator incorrectly auto-detects the cloud provider when using the cloud drive feature. This happens when you leave the spec.cloudStorage.provider
field blank and you're running on any of the following environments:
- Bare metal
- RKE2
- vSphere
If you encounter this issue, Portworx will fail to come up successfully during a fresh install or update.
As a workaround, please update the spec.cloudStorage.provider
field in your StorageCluster spec to the correct value for your deployment. Correct values are:
pure
vsphere
aws
azure
gce
ibm
oracle
csi
Improvements
In this release, Portworx Operator provides improvements in the following areas:
Issue Number | Issue Description |
---|---|
PWX-32147 | The Portworx Operator now supports the OpenShift dynamic console plugin for air-gapped clusters. |
PWX-31732 | Users can now list all the pods deployed by the Portworx Operator using the filter: operator.libopenstorage.org/managed-by=portworx . |
PWX-29299 | Users can now install Portworx on Oracle Kubernetes Engine (OKE) clusters with Portworx Operator Helm charts. |
PWX-25748 | Users can now install Grafana and the associated Portworx dashboards with spec.monitoring.grafana.enabled |
Fixes
Portworx Operator has resolved the following issues in this release:
Issue Number | Issue Description |
---|---|
PWX-34239 | The Portworx Operator caused PVC provisioning issues in GKE environments due to a missing GKE install chart file in Helm. Resolution: portworx.io/is-gke: "true" annotation is added to the elasticsearch-sc StorageClass spec to resolve this issue. |
PWX-33831 | Issue: A cordoned node may leave the Portworx kvdb-api pod in Failed state and such pods can interfere with the node's upgrade or maintenance.Resolution: The operator now explicitly checks for the kvdb-api pods in Failed state and explicitly cleans them up. |
PWX-31025 | Migration from a Portworx DaemonSet deployment to a Portworx Operator deployment sometimes became stuck in a perpetual Initializing state, even after successful completion of migration.Resolution: This issue is now resolved. |
PWX-30455 | The StorageCluster spec can be used to add custom container mounts. However it was not possible to use the same mounts as the default Portworx installation directories. Resolution: The custom mounts in the StorageCluster spec can now be used to override default Portworx directories. For example, the /opt/pwx or /var/cores mounts may be changed to a different directory on a partition with more disk-space. |
Known issue
Portworx Operator has the following known issue for this release:
Issue Number | Issue Description |
---|---|
PWX-32111 | PX-Security is not currently supported on PX-Store V2, and Operator pre-check will not proceed for this combination. |
23.7.0
September 12, 2023
Notes
- This release addresses security vulnerabilities to provide enhanced security.
- For air-gapped installs, the OCP Dynamic Plugin is in tech preview.
Portworx Operator offers complete support for generic HTTP/HTTPS proxies, but currently there is a limited support for HTTPS proxies using SSL inspection (for example, the Next-Generation Firewall that are re-encrypting the SSL traffic). Therefore to support HTTPS proxy with SSL:
- Portworx by PureStorage recommends to configure Portworx storage cluster, the Operator, and License Server similar to air-gapped environments
- For Portworx storage cluster, you have the option to configure proxy's self-signed Certification Authority (CA) certificate
Telemetry connection to Pure1 with the Next-Generation Firewall is not supported.
Portworx Operator offers the following improvements and fixes in this release:
Improvements
Portworx Operator has upgraded its functionalities in the following areas:
Issue Number | Issue Description |
---|---|
PWX-32188 | Portworx Operator now includes portworx.io/tls-cipher-suites and portworx.io/tls-min-version configuration parameters for portworx-pvc-controller to pass TLS cipher-suites preference and to set minimum TLS version respectively.Note: Due to golang limitation, selection of VersionTLS13 disables TLS cipher-suites customization. |
PWX-32147 | Portworx Operator enables OCP dynamic plugin installation for air-gapped clusters along with custom image registry support. In addition, PX-Security enabled clusters support OCP dynamic plugin. |
PWX-32011 | Portworx Operator can now leverage the PX_HTTPS_PROXY environment variable to configure the envoy to use internal URL to connect to the destination host. |
PWX-30520 | Portworx Operator offers enhanced security for JWT package. |
PWX-27765 | Storage cluster now displays defined and detailed phases of install and upgrade such as initializing , running , degraded , uninstalling and so on in the condition list. |
Fixes
Portworx Operator has resolved the following issues in this release:
Issue Number | Issue Description |
---|---|
PWX-32145 | Issue: OCP dynamic plugin images were not listed in px-versions ConfigMap.Resolution: This issue is now fixed. |
PWX-31944 | Issue: In air-gapped environments, spin up of csi-ext-pod pod used to fail because of csi-health-monitor-controller container included in this pod by default.Resolution: csi-health-monitor-controller container is disabled now to spin up csi-ext-pod without interruptions. |
PWX-31915 | Issue: A cordoned node may leave the Portworx kvdb-api pod in Completed state sometimes and such pods can interfere with the node's upgrade or maintenance.Resolution: The operator now explicitly checks for the kvdb-api pods in Completed state and eliminates the terminated pods. |
PWX-31842 | Issue: On PKS platform, restart of Portworx pods and service used to cause excessive mounts, slow down IO operations until the host became unresponsive. Resolution: Upgrade the operator version to 23.7.x and reboot the affected PKS nodes. |
23.5.1
July 11, 2023
Fixes
Issue Number | Issue Description |
---|---|
PWX-32051 | Issue: Port used by telemetry could also be configured as NFS port on certain distributions. Resolution: Portworx Operator picks 9029 as the new port for telemetry envoy from Portworx 3.0.0 and above versions. |
PWX-32073 | Issue: CSI provisioner issued multiple requests for PVC provisioning causing system delays. Resolution: CSI provisioner timeout is updated to avoid delays. |
PWX-32197 | Issue: The required config section was missing for proxied Envoy versions greater than 1.22. Without this config, telemetry pods could not start. Resolution: The required config section for Envoy running with HTTP proxy has been updated, and telemetry pods now start as expected. |
23.5.0
June 13, 2023
New features
Portworx by Pure Storage is proud to introduce the following new features:OpenShift users on OCP 4.12 and higher versions can now enable the Console plugin option during Portworx Operator installation or upgrade to use the Portworx Cluster dashboard within the OpenShift UI to manage their Portworx cluster. This avoids switching and navigating between different management interfaces to perform Day 2 operations.
Portworx Operator now supports loading installation images into multiple custom registries for seamless Portworx installation for all Kubernetes installation environments (air gapped and non-air gapped). You need to update the path of the custom registry for each of these components in the version manifest prior to installation. For more information, see Install Portworx on Kubernetes with a custom container registry.
Note
PodSecurityPolicy
resource is deprecated from Kubernetes 1.21 and unsupported from 1.25.x. Hence, you need to use either Pod Security Admission or third-party admission-plugin or both to impose restrictions on pods.
Improvements
Portworx Operator has upgraded its functionalities in the following areas:
Improvement Number | Improvement Description |
---|---|
PWX-26156 | If your Portworx runs on Kubernetes version 1.26 and higher, the operator auto-enables CSI in StorageCluster for both fresh install and upgrade to ease volume migration. |
PWX-27920 | Portworx Operator enables batching in metrics collector to reduce memory usage on large scale clusters. |
Fixes
Issue Number | Issue Description |
---|---|
PWX-28650 | Issue: Portworx Operator used to allow Autopilot to access all resources on the cluster without any restrictions. Resolution: The operator now enables Autopilot to provide selective access for the required resources (actions), thus minimizing the RBAC permissions. |
PWX-30386 | Issue: StorageCluster warnings were issued when Portworx Operator tried to delete a non-existent component. Resolution: This issue is now fixed. |
PWX-30737 | Issue: Stork scheduler pods were not created and px-csi-ext pods were stuck in a pending state. Resolution: Update Stork scheduler ClusterRole to use Portworx SCC to create a Stork scheduler pod. |
PWX-30943 | Issue: Prometheus pod crashed due to out of memory issue with default memory and CPU. Resolution: Users can now control and enforce memory usage on their actual cluster deployment and set the required limit by editing the StorageCluster spec. |
PWX-31551 | Issue: The latest OpenShift installations have introduced more restrictive SELinux policies, as a result, non-privileged pods cannot access the csi.sock CSI interface file.Resolution: All Portworx CSI pods now run as privileged pods. |
Known issues (Errata)
Issue Number | Issue Description |
---|---|
PD-2156 | On OpenShift clusters running with Dynatrace, Portworx might not start after upgrading OCP version from 4.11 to 4.12 Workaround: Delete the Portworx pod on the node. |
23.4.1
September 1, 2023
Portworx Operator provides the following fix in this release:
Fix
Issue Number | Issue Description |
---|---|
PWX-32073 | Issue: During PVC provisioning, CSI CreateVolume function used to constantly retry volume creation causing CSI provisioner to become unresponsive leading to delays.Resolution: CSI provisioner timeout value is updated from 10 seconds to 5 minutes to provide adequate time for volume creation. |
23.4.0
May 03, 2023
Improvements
Portworx Operator has enhanced its functionalities in the following areas:
Improvement Number | Improvement Description |
---|---|
PWX-27168 | A new annotation is added to Portworx Operator to let users customize Prometheus alert rules without Portworx operator rolling back the change. Add the following annotations in the Prometheus spec to customize the alerts:metadata: annotations: operator.libopenstorage.org/reconcile: "0" |
PWX-24897 | You can configure Prometheus with these new flags in your StorageCluster spec for monitoring Portworx:
|
Fixes
Issue Number | Issue Description |
---|---|
PWX-29409 | Issue: In a cluster, if there was a zone with no nodes available for Portworx, Operator failed to pick a default value for the MaxStorageNodesPerZone parameter.Resolution: Operator now ignores zone(s) with no nodes and utilizes other nodes to calculate MaxStorageNodesPerZone parameter value. |
PWX-29398 | Issue: Operator triggered nil panic error if no nodes were available to install Portworx.Resolution: User interface displays an appropriate error message instead of panicking when there are no nodes available for Portworx installation. |
23.3.1
March 29, 2023
Improvements
Portworx Operator has upgraded or enhanced functionality in the following areas:
Improvement Number | Improvement Description |
---|---|
PWX-30005 | Improvement for air-gapped clusters: The Operator now checks for Pure1 connectivity when enabled for the first time. If a telemetry cert has not yet been created and Portworx cannot reach Pure1, the Operator disables telemetry. |
23.3.0
March 22, 2023
Notes
- Starting with 23.3.0, the naming scheme for Operator releases has changed. Release numbers are now based on the year and month of the release.
- You must upgrade to Operator 23.3.0 to avoid an
ImagePullError
after April 3rd due to changes in the Kubernetes registry path. Kubernetes is freezinggcr.k8s.io
and moving to theregistry.k8s.io
repository on 3rd of April. For more information, see the Kubernetes blog.
New features
Portworx by Pure Storage is proud to introduce the following new features:telemetry is now enabled by default.
On fresh installations, all clusters will have telemetry enabled when you generate a spec from PX-Central.
When you upgrade the Portworx Operator to version 23.3.0, telemetry will be enabled by default unless you disable telemetry in the StorageCluster spec, or when the
PX_HTTPS_PROXY
variable is configured. :::note NOTE: For air-gapped clusters, you must disable telemetry explicitly during spec generation.
To learn how to disable telemetry, see the air-gapped installation section.
If you do not disable telemetry, telemetry pods will remain in theinit
state as Portworx fails to reach the Pure1 telemetry endpoint. This does not impact Portworx pods.:::
The StorageCluster spec for configuring Prometheus now contains the following new fields:
spec.Monitoring.Prometheus.Resources
: Provides the ability to configure Prometheus resource usage, such as memory and CPU usage. If the resources field is not configured, default limits will be set to CPU 1, memory 800M, and ephemeral storage 5G.spec.Monitoring.Prometheus.securityContext.runAsNonRootin
: Provides the ability to configure the Prometheus service type, and the default value is set totrue
.
Added a new environment variable
KUBELET_DIR
. This variable can be used to specify a customkubelet
directory path.Added an annotation
portworx.io/scc-priority
to the StorageCluster spec for configuring the priority of Portworx security context constraints (SCC).
Improvements
Portworx Operator has upgraded or enhanced functionality in the following areas:
Improvement Number | Improvement Description |
---|---|
PWX-28147 | When upgrading to Operator version 23.3.0, all CSI sidecar images will be updated to the latest versions. |
PWX-28077 | Operator will now update Prometheus and Alertmanager CRDs. |
Fixes
Issue Number | Issue Description |
---|---|
PWX-28343 | During the Operator upgrade, the old telemetry registration pod were not being deleted. Resolution: Changed the update deployment strategy of px-telemetry-registration to Recreate . Now the old pods will be deleted before the new ones are created. |
PWX-29531 | The prometheus-px-prometheus pods were not being created in OpenShift due to failed SCC validation.Resolution: This issue has been fixed. |
PWX-29565 | Upgrading OpenShift from version 4.11.x to 4.12.3 was failing for the Portworx cluster. Resolution: Changed Portworx SCC default priority to nil . |
PWX-28101 | If the kubelet path was not set to the default path, the CSI driver would fail to start, and the PVC could not be provisioned.Resolution: Now the KUBELET_DIR environment variable can be used to specify a custom path for the CSI driver. |
1.10.5
March 07, 2023
Updates
- Added the new
spec.updatestrategy.rollingupdate.minreadyseconds
flag. During rolling updates, this flag will wait for all pods to be ready for at leastminReadySeconds
before updating the next batch of pods, where the size of the pod batch is specified through thespec.updateStrategy.rollingUpdate.maxUnavailable
flag.
1.10.4
February 22, 2023
Updates
- Added a new annotation
portworx.io/is-oke=true
to the StorageCluster spec to support Portworx deployment on the Oracle Container Engine for Kubernetes (OKE) cluster.
Bug fixes
Fixed a bug where the Portworx PVC controller leader election resources conflicted with the resources used by the Kubernetes controller manager.
Fixed the Anthos Telemetry installation failure. Operator now allows two sidecar containers to run on the same node.
1.10.3
January 27, 2023
Bug fixes
- In Operator version 1.10.2, the Portworx pod was being scheduled on a random node because of a missing node name in the Portworx pod template. This issue is fixed in Operator version 1.10.3.
1.10.2
January 24, 2023
Updates
- Stork now uses
KubeSchedulerConfiguration
for Kubernetes version 1.23 or newer, so that pods are evenly distributed across all nodes in your cluster.
1.10.1
Dec 5, 2022
Updates
Added support for Kubernetes version 1.25, which includes:
Removed
PodSecurityPolicy
when deploying Portworx with Operator.Upgraded the API version of
PodDisruptionBudget
from policy/v1beta1 to policy/v1
Added a UI option in the spec generator to configure Kubernetes version when you choose to deploy Portworx version 2.12.
Operator is now deployed without verbose log by default. To enable it, add the
--verbose
argument to the Operator deployment.For CSI deployment, the px-csi-ext pods now set Stork as a scheduler in the px-csi-ext deployment spec.
Operator now chooses
maxStorageNodesPerZone
’s default value to efficiently manage the number of storage nodes in a cluster. For more details, see Manage the number of storage nodes.
1.10.0
Oct 24, 2022
Notes
IMPORTANT: To enable telemetry for DaemonSet-based Portworx installations, you must migrate to an Operator-based installation, then upgrade to Portworx version 2.12 before enabling Pure1 integration. For more details, see this document.
Updates
- Pure1 integration has been re-architected to be more robust and use less memory. It is supported on Portworx version 2.12 clusters deployed with Operator version 1.10.
- To reduce memory usage, added a new argument
disable-cache-for
to disable Kubernetes objects from controller runtime cache. For example,--disable-cache-for="Event,ConfigMap,Pod,PersistentVolume,PersistentVolumeClaim"
. - Operator now blocks Portworx installation if Portworx is uninstalled without a wipe and then reinstalled with a different name.
- For a new installation, Operator now sets the max number of storage nodes per zone, so that the 3 storage nodes in the entire cluster are uniformly spread across zones.
Bug fixes
- Fixed a bug where DaemonSet migration was failing if the Portworx cluster ID was too long.
1.9.1
Sep 8, 2022
Updates
- Added support for Kubernetes version 1.24:
- Added
docker.io
prefix for component images deployed by Operator. - To determine Kubernetes master nodes, Operator now uses the
control-plane
node role instead ofmaster
.
- Added
Bug Fixes
- In Operator 1.9.0, when you enabled the CSI snapshot controller explicitly in the StorageCluster, the
csi-snapshot-controller
sidecar containers might have been removed during an upgrade or restart operation. This issue is fixed in Operator 1.9.1.
1.9.0
Aug 1, 2022
Updates
- Daemonset to Operator migration is now Generally Available. This includes the following features:
- The ability to perform a dry run of the migration
- Migration for generic helm chart from Daemonset to the Operator
- Support for the
OnDelete
migration strategy - Support for various configurations such as external KVDB, custom volumes, environment variables, service type, and annotations
- You can now use the
generic helm chart
to install Portworx with the Operator. Note: Only AWS EKS has been validated for cloud deployments. - Support for enabling
pprof
in order to get Portworx Operator container profiles for memory, CPU, and so on. - The Operator now creates example CSI storage classes.
- The Operator now enables the CSI snapshot controller by default on Kubernetes 1.17 and newer.
Bug Fixes
- Fixed an issue where KVDB pods were repeatedly created when a pod was in the
evicted
oroutOfPods
status.
Known Issues
- When you upgrade Operator to version 1.9.0, the snapshot controller containers are removed from
px-csi-ext
deployment when theinstallSnapshotController
flag is set to true explicitly in the StorageCluster spec.
Workaround: To fix this issue, either restart Operator or upgrade to a newer version.
1.8.1
June 22, 2022
Updates
- Added support for Operator to run on IPv6 environment.
- You can now enable CSI topology feature by setting the
.Spec.CSI.Topology.Enabled
flag totrue
in the StorageCluster CRD, the default value isfalse
. The feature is only supported on FlashArray direct access volumes. - Operator now uses custom SecurityContextConstraints
portworx
instead ofprivileged
on OpenShift. - You can now add custom annotations to any service created by Operator.
- You can now configure
ServiceType
on any service created by Operator.
Bug Fixes
- Fixed pod recreation race condition during OCP upgrade by introducing exponential back-off to pod recreation when the
operator.libopenstorage.org/cordoned-restart-delay-secs
annotation is not set. - Fixed the incorrect CSI provisioner arguments when custom image registry path contains ":".
1.8.0
Apr 14, 2022
Updates
- Daemonset to operator migration is in Beta release.
- Added support for passing custom labels to Portworx API service from StorageCluster.
- Operator now enables the Autopilot component to communicate securely using tokens when PX-Security is enabled in the Portworx cluster.
- Added field
preserveFullCustomImageRegistry
in StorageCluster spec to preserve full image path when using custom image registry. - Operator now retrieves the version manifest through proxy if
PX_HTTP_PROXY
is configured. - Stork, Stork scheduler, CSI, and PVC controller pods are now deployed with
topologySpreadConstraints
to distribute pod replicas across Kubernetes failure domains. - Added support for installing health monitoring sidecars from StorageCluster.
- Added support for installing snapshot controller and CRD from StorageCluster.
- The feature gate for CSI is now deprecated and replaced by setting
spec.csi.enabled
in StorageCluster. - Added support to enable hostPID to Portworx pods using the annotation
portworx.io/host-pid="true"
in StorageCluster. - Operator now sets
fsGroupPolicy
in the CSIDriver object toFile
. Previously it was not set explicitly, and the default value wasReadWriteOnceWithFsType
. - Added
skip-resource
annotation to PX-Security Kubernetes secrets to skip backing them to the cloud. - Operator now sets the dnsPolicy of Portworx pod to
ClusterFirstWithHostNet
by default. - When using Cloud Storage, Operator validates that the node groups in StorageCluster use only one common label selector key across all node groups. It also validates that the value matches
spec.cloudStorage.nodePoolLabel
if a is present. If the value is not present, it automatically populates it with the value of the common label selector.
Bug Fixes
- Fixed Pod Disruption Budget issue blocking Openshift upgrade on Metro DR setup.
- Fixed Stork scheduler's pod anti-affinity by adding the label
name: stork-scheduler
to Stork scheduler deployments. - When a node level spec specifies a cloud storage configuration, we no longer set the cluster level default storage configuration. Before this fix, the node level cloud storage configuration would be overwritten.