Skip to main content

Portworx Operator Release Notes

25.2.2​

July 8, 2025

Improvements​

Improvement NumberImprovement Description
PWX-40116Portworx Operator now emits events on the StorageCluster object during Portworx and Kubernetes smart upgrades if a node is not selected for upgrade. Each event includes details explaining why the node was not selected for the upgrade.

Fixes​

Issue NumberIssue DescriptionSeverity
PWX-45078The Operator configured Prometheus to connect to KVDB metrics over HTTP, even when TLS was enabled. This caused the connections to fail.

User Impact: In clusters with KVDB TLS enabled, Prometheus could not scrape internal KVDB metrics. As a result, monitoring dashboards were incomplete and TLS handshake errors appeared in logs.

Resolution: The Operator now configures Prometheus to use HTTPS with TLS settings when KVDB TLS is enabled. This ensures that internal KVDB metrics are collected successfully.

Affected Versions: Versions 25.2.1		Minor

25.2.1​

June 23, 2025

note

We recommend upgrading to Portworx Operator version 25.2.1. Follow these guidelines for a seamless upgrade:

  • If you're currently running Portworx Operator version 24.2.4 or earlier, we recommend upgrading to Operator 25.2.1 directly by following the standard upgrade procedure.
  • If you're currently running Portworx Operator version 25.1.0 or 25.2.0, follow these steps:
    1. If you've labeled only the nodes that should run Portworx by using the px/enabled=true label, ensure that you've applied the workaround described here.
    2. Upgrade Portworx Operator.
  • On OpenShift clusters, if you're currently running Portworx Operator version 25.1.0, you cannot upgrade to 25.2.1 directly. You must uninstall Operator 25.1.0 (while retaining the StorageCluster and all operands), before installing the new version. For more information, see Upgrade notes for OpenShift.

Fixes​

Issue NumberIssue DescriptionSeverity
PWX-44096During a Portworx upgrade, if one or more nodes are cordoned, Portworx might upgrade multiple nodes simultaneously, exceeding the maxUnavailable configuration. This behavior occurs because of the default value of the cordon restart delay, which is set to 5 minutes.

User Impact: More nodes might be upgraded simultaneously than specified by maxUnavailable, exceeding the maxUnavailable limit.

Resolution: Now, when any nodes are cordoned, the operator overrides the value of the cordon restart delay to 0 seconds during Portworx upgrade. This ensures that only the number of nodes specified by maxUnavailable are upgraded at the same time.

Affected Versions: Versions 25.2.0 and earlier		Minor

Known issues (Errata)​

Issue NumberIssue Description
PWX-41729The Portworx Operator did not schedule Portworx pods on nodes labeled px/enabled=false. However, during node decommission, the label is set to px/enabled=remove before being changed to px/enabled=false after Portworx is fully removed. If removal takes time, the Operator could incorrectly reschedule Portworx pods.

User Impact: Portworx pods may be rescheduled on nodes that are in the process of being decommissioned, disrupting removal workflows.

Affected Versions: 25.2.1 and 24.2.4 and earlier

25.2.0​

May 25, 2025

note

Clusters running Portworx Operator version 24.2.4 can directly upgrade to 25.2.0 following the regular upgrades and skip upgrade to 25.1.0.

OpenShift Clusters running Portworx Operator version 25.1.0, won’t be able to upgrade to 25.2.0 directly. You must uninstall Operator 25.1.0(while retaining the StorageCluster and all operands), before installing the new version. For more information, see Upgrade notes for OpenShift.

New features​

  • Smart Upgrade now supports Kubernetes upgrades on additional platforms, expanding the streamlined and resilient upgrade process. Smart Upgrade maintains volume quorum and prevents application disruption during the parallel upgrade of Portworx and Kubernetes nodes.

    • Portworx upgrade: Smart Upgrade continues to be supported on all platforms.
    • Kubernetes upgrade: Smart Upgrade is now supported on the following platforms:
      • All OpenShift distributions
      • Google Anthos
      • Vanilla Kubernetes
      • Azure Kubernetes Service (AKS) (new)
      • Amazon Elastic Kubernetes Service (EKS) (new)
      • Google Kubernetes Engine (GKE) (new)

    For more information, see Smart Upgrade.

  • Custom labels are supported on all Portworx managed components. You can apply labels to each component or component type via StorageCluster.spec.metadata.labels to improve observability. For more information, see StorageCluster.

Improvements​

Improvement NumberImprovement Description
PWX-38655Portworx Operator now supports the spec.monitoring.telemetry.hostNetwork field in the StorageCluster YAML. This setting controls whether Telemetry Pods run using either the Kubernetes network (default) or the host network.
PWX-42823When runtime options are specified as part of the misc-args annotation using rt_opts, and also provided as key-value pairs under RuntimeOptions in the cluster or node specification, the key-value pairs in RuntimeOptions are ignored, and a warning is raised in the StorageCluster. For more information, see StorageCluster CRD reference.
PWX-42799Portworx Operator now supports configuring resource limits for Telemetry pods through the StorageCluster specification. You can now use spec.monitoring.telemetry.resources to set CPU and memory limits for the registration and phonehome pods.
PWX-41647The Portworx Operator no longer creates in-tree StorageClasses on Kubernetes 1.31 and later. Users should use Portworx CSI StorageClass for volume provisioning.
PWX-42375Portworx Operator now supports configuring miscArgs at the node level using the spec.nodes[i].miscArgs field, in addition to cluster-level configuration using portworx.io/misc-args annotation. If both are set, the two sets of arguments are combined. When a key exists in both, the node-level value takes precedence. If no arguments are specified at the node level, the cluster-level annotations apply to all PX pods.

Fixes​

Issue NumberIssue DescriptionSeverity
PWX-44295When upgrading the operator to version 25.1.0, the StorageCluster spec placement configuration was reset to default values. This caused Portworx pods to stop running on infrastructure nodes in OpenShift clusters that had custom placement rules.

User Impact: This issue affects OpenShift clusters where Portworx was explicitly configured to run on infrastructure nodes. As a result, workloads relying on Portworx storage on those nodes have become unavailable.

Resolution: The operator now preserves existing custom node placement configurations that include infrastructure nodes during upgrades. New installations will continue to use the default placement rules, which exclude infrastructure nodes.

Affected Versions: 25.1.0		Major
PWX-42315If you disable telemetry in the Portworx Operator configuration, the Operator deletes the associated telemetry ConfigMap. However, it doesn’t remove the corresponding telemetry volume mounts from Portworx pods.

User Impact: Portworx pods reference stale telemetry volume mounts. When a node reboots or a pod restarts, Portworx containers fail to start because the ConfigMap volumes are missing.

Resolution: The Portworx Operator now detects changes in the telemetry specification and triggers a Portworx upgrade to apply them. This ensures that Portworx and Portworx API pods on affected nodes restart with the correct volume mounts, preventing stale references and startup failures.

Affected Versions: 24.2.4 or earlier		Minor
PWX-41729The Portworx Operator did not schedule Portworx pods on nodes labeled px/enabled=false. However, during node decommission, the label is set to px/enabled=remove before being changed to px/enabled=false after Portworx is fully removed. If removal takes time, the Operator could incorrectly reschedule Portworx pods.

User Impact: Portworx pods may be rescheduled on nodes that are in the process of being decommissioned, disrupting removal workflows.

Resolution: The Operator now checks for both px/enabled=remove and px/enabled=false before scheduling Portworx pods. This prevents unintended scheduling during node decommissioning.

Note: When upgrading to operator version 25.1.0 from an earlier version, all operator-managed pods will restart—except Portworx oci-monitor pods. This is due to a new node affinity rule.

Affected Versions: 24.2.4 or earlier		Minor
PWX-41181Migration from the Portworx DaemonSet to the Portworx Operator failed when performed using Helm. Helm created the StorageCluster directly, which prevented the Operator from adding the required migration status condition. As a result, the Operator could not detect that migration was approved, even when the correct annotation was set.

User Impact: The migration could not proceed, and Portworx pods were not created by the Operator. This left the cluster in an incomplete state.

Resolution: The Operator now adds the migration status condition when migration starts, even if the StorageCluster was created outside the Operator. This allows migration to complete as expected.

Affected Versions: 24.2.4 or earlier		Minor
PWX-40283During Portworx upgrades, the StorageCluster status remained Running even after the upgrade started. This made it appear that the upgrade was complete when it was still in progress.

User Impact: Users could not track the upgrade accurately. The Running status caused confusion and limited visibility into progress.

Resolution: Portworx Operator now sets the status to Updating when the upgrade begins and reverts it to Running after completion. This provides clearer tracking of the upgrade process.

Affected Versions: 24.2.4 or earlier		Minor
PWX-37878STORK could not reschedule pods during node maintenance because it was unable to access the Portworx API. This triggered a "No node found with storage driver" error.

User Impact: Pods could become stuck or unavailable during node maintenance, affecting application availability.

Resolution: A liveness probe was added for the Portworx API. This allows STORK to detect when a node is down or in maintenance mode and reschedule pods appropriately.

Note: Portworx API pods will restart during the upgrade to operator version 25.1.0 due to the new liveness probe.

Affected Versions: 24.2.4 or earlier		Minor
PWX-36253When external etcd was configured with only the --cacert flag, the Portworx Operator did not add the required volume mount for the etcd secret. This caused Portworx pods to fail during DaemonSet to Operator migration.

User Impact: Clusters using external etcd with only a CA certificate failed to start Portworx pods during migration, blocking successful initialization.

Resolution: Portworx Operator now adds the etcd secret volume mount when only the --cacert flag is set. Portworx pods now start successfully during migration in this configuration.

Affected Versions: 24.2.4 or earlier		Minor

Known issues (Errata)​

Issue NumberIssue DescriptionSeverity
PWX-44331

To prevent Portworx from being installed and started on specific nodes, Portworx recommends labeling the nodes with px/enabled=false

kubectl label nodes <node-name> px/enabled=false --overwrite

If you labeled only the nodes where Portworx should run by using the px/enabled=true label, after upgrading to Portworx Operator version 25.2.0, Portworx pods might be scheduled on nodes that are not labeled as well.

User Impact:

After upgrading to Operator version 25.2.0:

  • Portworx pods are scheduled on unlabeled nodes.
  • The Operator reverts the custom nodeAffinity to the default in the StorageCluster every time it is changed.

Workaround:

Before upgrading to Operator version 25.2.0, update the nodeAffinity section of the StorageCluster specification to allow both true and enabled values for the px/enabled key:

spec:
  placement:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: px/enabled
            operator: In
            values:
              - "true"
              - "enabled"
Minor

Upgrade notes for OpenShift​

Portworx Operator version 25.1.0 has been replaced with 25.2.0. Upgrade to version 25.2.0.

note

These steps apply only to OpenShift clusters.
For other Kubernetes environments, refer to Upgrade Portworx Operator

Follow these steps to uninstall Operator version 25.1.0 and install version 25.2.0 on OpenShift clusters:

  1. In the OpenShift web console, go to Operators > Installed Operators.

  2. Verify that the installed version of Portworx Enterprise is 25.1.0.

  3. Select Actions > Uninstall Operator.

  4. In the confirmation dialog, clear the Delete all operand instances for this operator check box. This ensures that Portworx continues to run after uninstallation of Operator.

    Uninstall Operator

  5. Select Uninstall again to confirm.

  6. After the uninstallation completes, return to OperatorHub.

  7. Search for Portworx Enterprise, and then install version 25.2.0.

25.1.0​​

May 21, 2025

This release has been replaced with version 25.2.0.

important

If Portworx was running on nodes labeled node-role.kubernetes.io/infra before upgrading to Portworx Operator version 25.1.0, you might notice that Portworx no longer runs on those nodes after the upgrade. This issue occurs because upgrading the Portworx Operator to version 25.1.0 resets the pod placement configuration in the StorageCluster resource to its default values. As a result, Portworx pods may stop running on infrastructure nodes in OpenShift clusters that use custom placement rules.

Workaround: Manually edit the StorageCluster resource to remove the node affinity rule that excludes infrastructure nodes.

Delete the following section under the spec.placement.nodeAffinity field from the StorageCluster spec.

StorageCluster
- key: node-role.kubernetes.io/infra
  operator: DoesNotExist

24.2.4​

April 17, 2025

Improvements​

Improvement NumberImprovement Description
PWX-41562The portworx.io/preflight-check annotation now supports only skip and true as valid values. If set to true, the Operator does not modify the annotation value after the pre-flight check completes. The Operator now relies solely on pre-flight conditions to determine whether checks have been executed. If the annotation is not explicitly set in the StorageCluster specification, the Operator defaults to skip. However, for Amazon Elastic Kubernetes Service (EKS) clusters running Portworx Enterprise 3.0 or later, the default is true.

Notes:
  • Starting with Portworx Operator 24.2.4, the Operator automatically migrates older values of the portworx.io/preflight-check annotation during upgrades. In earlier versions, if this annotation was set to false, the Operator interpreted it as skip. During upgrade, the Operator now updates the annotation to true if a pre-flight check had previously been completed, or to skip if no check was ever performed. This migration ensures that the annotation accurately reflects the pre-flight check status.
  • Do not modify the portworx.io/preflight-check annotation during preflight execution, installation, or migration. Changing the annotation value while checks are still in progress can destabilize the cluster. If the annotation is toggled back to true after being changed mid-check, the pre-flight check may run again unnecessarily.

24.2.3​

March 6, 2025

note

Portworx now supports Kubernetes version 1.31, starting from version 1.31.6. Before upgrading Kubernetes to 1.31.6 or later, update the Portworx Operator to version 24.2.3.

Improvements​

Improvement NumberImprovement Description
PWX-41353Starting from Kubernetes 1.31, support for the in-tree provisioner kubernetes.io/portworx-volume has been removed. All storage requests are now redirected to the CSI driver pxd.portworx.com. Kubernetes automatically manages this transition by converting in-tree objects stored in etcd into CSI-compatible objects. This migration requires additional annotations on existing in-tree PersistentVolumes (PVs), which are handled by the Portworx Operator.

When security is enabled and guest access is disabled, the Portworx Operator annotates PVs with the authentication secret name and namespace. The secret name and namespace are retrieved from the StorageClass annotations openstorage.io/auth-secret-name and openstorage.io/auth-secret-namespace, respectively. If unavailable, the corresponding PersistentVolumeClaim (PVC) is checked. If no secret is found, the default px-user-token is used.

To comply with recent kubelet changes that restrict secret access to only the pods running on a node, the Portworx Operator now mounts user and admin secrets directly into portworx-api DaemonSet pods. This ensures that kubelet can access the necessary secrets. For additional secrets beyond the default px-user-token and px-admin-token, users should create a separate DaemonSet to mount and manage them.

Upgrade Considerations:
  • For a seamless transition, upgrade the Portworx Operator to version 24.2.3 before upgrading Kubernetes to version 1.31.
  • If you create in-tree persistent volumes (PVs) after upgrading the operator, the system does not automatically add the required annotations. To ensure proper annotations are added, restart the Portworx Operator pod.
  • In Portworx security-enabled clusters, upgrading to Operator 24.2.3 causes portworx-api pods to restart as new secrets are added.
  • Known limitation: Increasing the replica count of StatefulSets using in-tree PVs after upgrading Kubernetes does not create new PVs or trigger migration.

Note: If you have a disaster recovery setup, you must upgrade the Operator on both the source and destination clusters before upgrading Kubernetes to version 1.31.

Fixes​

Issue NumberIssue DescriptionSeverity
PWX-39594On some platforms, the Kubernetes cluster attempts to update the portworx-api DaemonSet and px-csi-ext Deployment with tolerations that are not specified in the StorageCluster. As a result, the Portworx operator, which is unaware of these tolerations, tries to remove them. This results in a loop where the portworx-api and px-csi pods keep restarting.

For example, if you enable the AlwaysPullImage admission plugin, it ensures that imagePullPolicy is set to Always for all Pods. If your StorageClass Template (STC) specifies imagePullPolicy: IfNotPresent, PX-Operator will continuously recreate the Pods to match the STC value, causing repeated restarts.
Similarly, if an external source modifies PX configurations—such as container or volume settings—restarts will occur.

User Impact: Continuous restarts of portworx-api and px-csi-ext pods due to the mismatched tolerations between the StorageCluster spec and the running Kubernetes resources.

Resolution: PX-Operator now compares the following configurations from the StorageCluster (STC) with the:
  • Currently running portworx-api DaemonSet and px-csi-ext Deployment:
    • Container and volume configurations
    • imagePullPolicy
    • Image Pull Secrets
    • Host Network
    • Tolerations
    • Service account name
    • Affinity settings
  • Complete container configuration of portworx-kvdb pods.

If PX-Operator detects any differences, it updates these resources to match the StorageCluster definition.
To avoid unintended restarts, users should ensure that any externally applied changes (such as Admission Plugins modifying tolerations) match the STC configuration.

Affected Versions: 24.1.2 to 24.2.2		Critical

Known issues (Errata)​

  • PWX-42235: When PX-Security is disabled in the StorageCluster, the operator initiates an upgrade to apply the change. During the upgrade, Portworx fails to mount secured persistent volumes (PVs), leading to VM live migration failures and blocking the upgrade.

    Workaround: Follow the steps below:

    1. Stop all pods consuming Portworx volumes before disabling security.
    2. Manually stop KubeVirt VMs before disabling security to prevent live migration failures.
    3. Restart the stopped VMs once all nodes have been successfully upgraded.

    Affected Versions: 24.2.3

    Severity: Minor

24.2.2​

February 3, 2025

Fixes​

Issue NumberIssue DescriptionSeverity
PWX-41581On the SUSE Linux Micro platform, the configuration directory was located on a managed partition. This setup inadvertently led to conflicts during configuration updates.

User Impact: The conflict could result in startup failures, potentially impacting system functionality and leading to service disruptions.

Resolution: The default configuration directory has been relocated to a more stable path, ensuring improved reliability and smoother configuration updates.

Affected Versions: 24.2.1 or earlier		Major

24.2.1​

January 6, 2025

Improvements​

Improvement NumberImprovement Description
PWX-40632When the Operator is upgraded to 24.2.1 in OpenShift Container Platform (OCP), kvdb pods will be restarted to allow the security context constraint (SCC) restriction to be added to those pods. The Operator ensures that only one kvdb pod gets restarted at a time. Sometimes, due to race conditions, two kvdb pods might restart with a very short interval between them. However, they come up within a few seconds.
PWX-40424The Operator performs a health check during installation to verify if the Kubernetes version is supported for the installed PX version on specific platforms. If the version is unsupported, the Operator raises a warning but proceeds with the installation. The supported platforms for the health check are GKE, AKS, EKS, IKS, OKE, RKE2, OCP, ROKS, ROSA, ARO, and MKE. For other platforms, the health check is skipped.
PWX-38148Updated the following packages to resolve security vulnerabilities:
  • k8s.io
  • gRPC
  • golang

Bug Fixes​

Issue NumberIssue DescriptionSeverity
PWX-40405In certain scenarios, the order of runtime options (rt_opts) in the Portworx container caused an unexpected restart of OCI monitor pods, impacting other functions.

User Impact: Unexpected restarts of OCI monitor pods could disrupt other services and functions dependent on these pods.

Resolution: The order of runtime options (rt_options) for the px service no longer causes OCI monitor pods to restart.

Affected Versions: 24.2.0 or earlier		Minor
PWX-39891Some pods, like Autopilot and Stork Scheduler, that use the portworx-restricted SCC might not work because the portworx-restricted SCC does not allow CSI volumes.

User Impact: Pods using the portworx-restricted SCC might fail to start or function correctly.

Resolution: OCP supports CSI inline volumes only from version 4.13. Therefore, the portworx-restricted SCC for OCP versions 4.13 and later will allow the usage of CSI volumes. Portworx by Pure Storage recommends upgrading OCP cluster and Operator to the latest version.

Affected Versions: 24.2.0 or earlier running on OCP 4.12 or earlier		Minor
PWX-37637On OCP, the Operator did not set the required SCC for all components, making them rely on OCP-assigned SCCs. This might cause issues if the assigned SCCs do not allow the pods to work properly.

User Impact: If OCP-assigned SCCs are incorrect, it might cause components to not work as expected.

Resolution: All component pods in the Portworx cluster in OCP now have a required SCC annotation added to them. This ensures that no other SCC is assigned by OCP, causing the pods to function as expected. The components and their required SCCs are as follows:
  • kvdb: portworx-restricted SCC
  • csi: portworx SCC
  • telemetry-registration: portworx SCC
  • telemetry-phonehome: portworx SCC
  • px-plugin-proxy: restricted v2 SCC
  • px-plugin: restricted v2 SCC
  • portworx api: portworx SCC
  • autopilot: portworx-restricted SCC
Note: These pods will restart as part of the Operator upgrade.

Affected Versions: 24.2.0 or earlier		Major

24.2.0​

December 2, 2024

note

If you have installed Portworx with Pure Storage FlashArray configured with cloud drives on the vSphere platform, Portworx pods will restart when you upgrade the Operator from earlier versions to 24.2.0; this is an expected behavior.

New features​

Portworx by Pure Storage is proud to introduce the following new features:

  • Smart upgrade feature introduces a streamlined, resilient upgrade process for Portworx and Kubernetes nodes, allowing them to be upgraded in parallel while maintaining volume quorum and without application disruption.

    • Portworx upgrade: Smart upgrade is supported on all platforms.
    • Kubernetes upgrade: Smart upgrade is supported only on the following platforms:
      • All OpenShift Platforms
      • Google Anthos
      • Vanilla Kubernetes

  • Health checks determines if the target system meets the requirements for Portworx before installation. It provides pass, fail, or warning notifications and assists you in resolving the issues right before the installation of Portworx.

Improvements​

Issue NumberIssue Description
PWX-34825When the version configmap is not available or the URL is not reachable, the installation fails with an appropriate error instead of falling back to default images.
PWX-37782Upgraded the CSI images to address security vulnerabilities.
  • CSI Provisioner
  • Node Driver Registrar
  • CSI Resizer
  • Snapshot Controller
PWX-38103If multiple storageless nodes have the same scheduler node name and one of them is online, the offline nodes are ignored. This ensures that upgrade is not blocked because of an offline node, which anyway gets auto-decommissioned in some time.
PWX-39104We have optimized the Portworx Operator to reduce memory consumption on large clusters.

Fixes​

Issue NumberIssue DescriptionSeverity
PWX-36251When uninstalling the Portworx with UninstallAndWipe delete strategy, the following cloud drive ConfigMaps are not deleted.

User Impact: Users were not able to uninstall Portworx completely.

Resolution: The Portworx Operator now deletes these cloud drive ConfigMaps as well.

Affected Versions: 24.1.3 and earlier		Minor
PWX-38509Provisioning of KubeVirt VM fails if the bootOrder is not specified for the VM disks and the first disk is not a PVC or a DataVolume.

User Impact: Portworx upgrade gets stuck as storage nodes become unavailable.

Affected Versions: 24.1.0, 24.1.1, 24.1.2, and 24.1.3		Minor

Known issues (Errata)​

  • PWX-39594: On MKE platforms, the Kubernetes cluster attempts to update the portworx-api and px-csi pods with tolerations that are not specified in the StorageCluster. As a result, the Portworx operator, which is unaware of these tolerations, tries to remove them. This results in a loop where the portworx-api and px-csi pods keep restarting.

    Workaround: Follow the steps below:

    1. Ensure that jq or yq installed on your machine.

    2. Get the kubeconfig of your cluster.

    3. Get taints on the cluster nodes.

      • For jq, run the kubectl --kubeconfig=<path/to/kubeconfig> get nodes -ojson | jq -r . | jq -r '.items[].spec.taints' command.

      • For yq, run the kubectl --kubeconfig=<path/to/kubeconfig> get nodes -oyaml | yq -r . | yq -r '.items[].spec.taints' command.

      The command output might look like the sample below:

      null
      null
      null
      [{"effect": "NoSchedule", "key": "com.docker.ucp.manager"}]
      null
      null

    4. Apply the tolerations to the StorageCluster based on the command output in the previous step.

      For example:

      spec:
        placement:
          tolerations:
          - key: com.docker.ucp.manager
            operator: Exists

    Affected Versions: 24.1.2 and 24.1.3

    Severity: Critical

24.1.3​

October 29, 2024

Fixes​

Issue NumberIssue DescriptionSeverity
PWX-36976Kubernetes typically provides credentials for pods to access the Kubernetes API, which are automatically mounted in the /run/secrets/kubernetes.io/serviceaccount/token file. However, in recent versions of Kubernetes, the service account token's lifetime is tied to the lifetime of the Portworx pod. As a result, the service account token that the Portworx service uses becomes invalid when a Portworx pod terminates and all Kubernetes API calls fail with unauthorized errors, which can be seen in the Portworx logs. This issue can cause Kubernetes upgrades to get stuck while waiting for application pods to terminate.

User Impact: Kubernetes platform upgrades may fail due to failure to evict the application pods.

Resolution: Operator 24.1.3 creates a new token for Portworx version 3.2.0, which is periodically refreshed for use by Portworx. Note: Portworx versions prior to 3.2.0 will ignore the new token and continue to work as before.

Affected Versions: 24.1.0, 24.1.1, and 24.1.2		Minor

24.1.2​

October 11, 2024

Fixes​

Issue NumberIssue DescriptionSeverity
PWX-38082Portworx CSI pods are crashing on clusters running on RHEL 9.x Linux nodes when SELinux is enabled.

User Impact: The csi-node-driver-registrar is not starting, causing application pods to fail to use Portworx PVCs.

Resolution: Permissions were corrected in the csi-registrar container, allowing Portworx PVC volumes to work correctly on RHEL 9.x Linux nodes with SELinux enabled.

Affected Versions: 24.1.0 and 24.1.1		Major
PWX-36650OpenShift applies Security Context Constraints (SCC) to the Portworx Operator pod based on its need and requirement. However, when a new SCC with lower privileges than the "anyuid" SCC was added to the cluster, the new SCC would be applied instead, causing the Portworx Operator to fail to start.

User Impact: The Portworx Operator did not come up due to the CreateContainerConfigError.

Resolution: The Operator now adds an openshift.io/required-scc annotation to the Operator and Stork pods to specify which SCC should be applied. The Operator and Stork pods will be assigned the "anyuid" SCC, while the Stork-scheduler pod will be assigned the portworx-restricted SCC.
Note: This fix is applicable on OCP version 4.14 or later. If you are running OCP 4.13 or earlier versions, Portworx by Pure Storage recommends elevating the SCC's permission based on the pod.

Affected Versions: 24.1.1 and earlier		Major

24.1.1​

July 25, 2024

note
  • The Portworx Operator now supports the fresh installation of Portworx on OCP 4.16 clusters. For official OCP upgrade path from OCP 4.15 to OCP 4.16, you can refer to the Red Hat OpenShift documentation.

Improvements​

Issue NumberIssue Description
PWX-37545When you set the autoUpdateStrategy to Always or Once and update the px-versions ConfigMap, the system will now automatically update the Prometheus Operator, Alertmanager, and Grafana images based on the px-versions ConfigMap.
PWX-37634We've optimized the startup time for telemetry pods. Previously, the px-telemetry-register configmap didn't set the internalHostnameLookupRetryCount, causing the telemetry registration pod to retry the DNS lookup of the internal hostname up to 30 times. Now, we've set internalHostnameLookupRetryCount to 3, reducing startup time when the internal hostname is unreachable.
PWX-32229We've enhanced the storage configuration validation. Now, configuring both local and cloud storage at the cluster level or node level is not allowed. If such a configuration is detected, the StorageCluster will transition to a degraded state and not proceed with the deployment until the Spec is corrected.
PWX-36928We've added explicit validation of Telemetry SSL certificates. Now, the Telemetry SSL certificates are validated against the Cluster UUID during the Telemetry pod startup.
PWX-30050We've added an optional field priorityClassName to the StorageCluster for specifying priority classes. You can now assign a priority class to Portworx pods through the StorageCluster configuration.

Fixes​

Issue NumberIssue Description
PWX-37852The Telemetry client SSL certificate expiration led to errors during the re-registration of the cluster with the Telemetry backend. Additionally, attempts to manually delete the client SSL certificate and clean up the status on Skyline also failed

User Impact: Errors occurred when trying to determine the appliance ID (Portworx UUID), leading to disruptions in the telemetry service.

Resolution: The Portworx Operator is now passing the Portworx UUID directly to Telemetry. This simplifies the discovery of the Portworx UUID and reduces the possibility of errors.

Affected Versions: All

24.1.0​

June 11, 2024

note
  • The PodDisruptionBudget (PDB) improvements are available with Portworx Enterprise version 3.1 or later.
  • In GKE environments, if you override the default PDB value, ensure that the maxUnavailable is less than the storage-pdb-min-available value for a balanced speed and disruption. Portworx by Pure Storage recommends using the following configurations for surge upgrades:
    • maxSurge=1
    • maxUnavailable=0
  • In OpenShift platforms, if you override the default PDB value, ensure that the storage-pdb-min-available value is greater than or equal to OCP's MCP maxUnavailable.

Improvements​

Issue NumberIssue Description
PWX-36663The px-prometheus account user is added to the Portworx Security Context Constraint only in either of the following cases:
  • The platform version is below OpenShift 4.12.
  • Portworx Prometheus monitoring is explicitly enabled using the spec.monitoring.prometheus.enabled flag for all supported Openshift versions.
PWX-35633When you override the PDB value using the portworx.io/storage-pdb-min-available annotation, if the minAvailable value is greater than or equal to the number of storage nodes, Portworx uses the default value for PDB.
PWX-36258When you override the PDB value using the portworx.io/storage-pdb-min-available annotation, if the minAvailable value is less than the quorum number of storage nodes, Portworx uses the default value for PDB.
PWX-35418    The total storage node count is set to the number of Portworx storage nodes. Therefore, any Portworx node that is not part of the Kubernetes cluster is also considered in the calculation of numStorageNodes. This ensures that such nodes are not down during the upgrade, as it can cause two Portworx nodes to go down at the same time, losing volume quorum.
PWX-34410The calculation of the numStorageNodes value for DR clusters now includes all storage nodes in the current cluster domain, even those that are not part of the current Kubernetes cluster.
PWX-34271For KubeVirt environments, when you upgrade the Portworx on a node, the Portworx Operator evicts the KubeVirt VMs by live-migrating them before upgrading the Portworx on that node. For more information, see Manage storage for KubeVirt VMs.
PWX-25339The Portworx cluster upgrade ensures that the internal KVDB node does not lose quorum. When a KVDB node is down, the remaining KVDB nodes will not be upgraded until the node that went down is back online, ensuring that only one KVDB node is offline at a time.

Fixes​

Issue NumberIssue Description
PWX-36551Plugin images were not updated even after the Portworx version in ConfigMap was updated.

Resolution: Image versions for the dynamic plugin and dynamic plugin proxy can now be updated with changes in the ConfigMap versions and endpoint when autoUpdateComponents is set to Once or Always in the StorageCluster.
PWX-36364While upgrading the Kubernetes cluster in the EKS environment, the kube-scheduler-amd64 and kube-controller-manager-amd64 images were not automatically upgraded to match the Kubernetes cluster version.

Resolution: After upgrading the Kubernetes cluster, the kube-scheduler-amd64 and kube-controller-manager-amd64 images are also upgraded to match the Kubernetes cluster version, and the pods are updated.
PWX-36298In an OpenShift environment, the master nodes were not considered while calculating the value of maxStorageNodesPerZone, which resulted in a number of nodes equal to the number of master nodes coming up as storageless nodes.

Resolution: When an annotation to run Portworx on master nodes is set in StorageCluster, the operator considers this flag while setting the value of maxStorageNodesPerZone. Thus, all master and worker nodes will be storage nodes unless explicitly specified otherwise.
PWX-36159The PVC controller failed to come up on the K3s cluster because default ports 10252 and 10257 for PVC were already in use.

Resolution: The default configuration is changed and uses the following port numbers on K3s deployment:
  • Insecure port: 10260
  • Secure port: 10261
PWX-31625In OpenShift environments, Portworx created excess portworx-pvc-controller pods, which caused warnings in VMware and vSphere environments.

Resolution: Portworx no longer creates portworx-pvc-controller pods in OpenShift environments.
PWX-28442    The Portworx Operator incorrectly calculated the minAvailable value in the PDB. This could lead to the downtime of an extra portworx node when it is not supposed to go down. In a 3–4 node cluster, this could also lead to a loss of cluster quorum.

Resolution: Now, the Portworx Operator looks at the NonQuorumMember flag from the node object, which always gives the correct value, even during the node initialization.

23.10.5​

April 5, 2024

note

Portworx Operator now supports installation of Portworx on OCP 4.15 clusters.

Improvement​

Issue NumberIssue Description
PWX-36460Portworx Operator adds a new annotation openshift.io/required-scc on Portworx pods to ensure that OpenShift assigns the correct Security Context Constraints (SCC) and the Portworx pods have the required permissions.

23.10.4​

March 19, 2024

Improvements​

Improvement NumberImprovement Description
PWX-36274For clusters on ARO, AWS, Azure, GCP, ROSA, ROKS or vSphere with OCP version 4.12 and above, Portworx Operator version 23.10.4 facilitates both the export of metrics to OCP Prometheus and the reading of metrics from it.
PWX-36198Portworx Operator 23.10.4 and Autopilot 1.3.14 enable you to deploy a RedHat Openshift Kubernetes Service (ROKS) cluster on IBM cloud using OCP 4.12 or later versions to view metrics in OpenShift Prometheus.

Fixes​

Portworx by PureStorage has fixed the following issues in this release:

Issue NumberIssue Description
PWX-35477Portworx deployments on OpenShift 4.14 and later no longer support the Prometheus deployment packaged with the Portworx Operator. This prevented upgrades for users wishing to upgrade from OCP 4.13.x to 4.14.x.

Resolution: You can now disable the Prometheus deployment packaged with the operator and use the OpenShift Prometheus stack for monitoring Portworx metrics, allowing you to upgrade to OpenShift 4.14 and later. This approach requires you to expose Portworx metrics to the OpenShift environment to ensure seamless monitoring integration.
PWX-35655A race condition sometimes deregistered the Portworx CSI driver immediately after initial registration. As a result, Portworx CSI volumes became stuck in the Pending state.

Resolution: This race condition and accidental de-registration no longer occurs. This change is applicable to Portworx versions 2.13 and above that have enabled CSI.
PWX-35008When the user disabled Stork component via StorageCluster, kube-scheduler image was not getting removed from the cache. As a consequence, once set, user could not update or override the kube-scheduler image path and/or version tag.

Resolution: If the user disables Stork component now, kube-scheduler component is cleanly removed and the user can update or override the kube-scheduler image and/or version tag.
PWX-35007User could not update or override the kube-scheduler and kube-controller-manager image paths or version tags using the px-versions ConfigMap.

Resolution: User can now update or override the kube-scheduler and kube-controller-manager image paths or version tags using the px-versions ConfigMap.
PWX-34360The csi-node-driver-registrar container ran inside the portworx pods. During a Kubernetes upgrade, when a node is cordoned, the portworx pod is drained. This de-registered the Portworx CSI driver as the container it was in was removed. Any application pod on that node that had not drained until this point became stuck in the Terminating state.

Resolution: The stability and availability of the csi driver during upgrades and maintenance processes is now improved, and the csi-node-driver-registrar container is no longer affected by node draining. This change is applicable to Portworx versions 2.13 and above that have enabled CSI.
PWX-35282Previously, when a Kubernetes node was cordoned, the Portworx pod was automatically deleted but would restart after a brief five-second interval. This behavior was intended to allow applications using Portworx volumes sufficient time to drain properly.

Resolution: In this release, when a node is cordoned prior to a drain, the Portworx pod will now be deleted and will not restart for the next five minutes, instead of the previous five seconds. This change is implemented following the movement of the csi-node-driver-registrar to the portworx-api daemonset, rendering the portworx pod unnecessary for this duration. This revised approach ensures more stability and less frequent disruptions in Portworx volumes management during node maintenance.

23.10.3​

February 08, 2024

note
  • The Operator version 23.10.3 is available only on OpenShift OperatorHub. Non-OpenShift customers are advised not to upgrade to this version and should await the release of version 23.10.4, which will be available shortly.
  • In order to use Portworx with OpenShift version 4.14, you must use Operator version 23.10.3 and Autopilot version 1.3.13. Visit the Upgrade OpenShift to version 4.14 with Portworx document for information on upgrading OpenShift to version 4.14 with Portworx.

Improvements​

Improvement NumberImprovement Description
PWX-36015On fresh installs on OpenShift 4.14+, the Portworx Operator will disable the monitoring.prometheus.enabled flag with a warning event. This is done to prevent users from accidentally enabling the unsupported Portworx managed prometheus deployment.

Fixes​

Portworx by PureStorage has fixed the following issues in this release:

Issue NumberIssue Description
PWX-35477Portworx deployments on OpenShift 4.14 and later no longer support the Prometheus deployment packaged with the Portworx Operator. This prevented upgrades for users wishing to upgrade from OCP 4.13.x to 4.14.x.

Resolution: You can now disable the Prometheus deployment packaged with the operator and use the OpenShift Prometheus stack for monitoring Portworx metrics, allowing you to upgrade to OpenShift 4.14 and later. This approach requires you to expose Portworx metrics to the OpenShift environment to ensure seamless monitoring integration.
PWX-35655A race condition sometimes deregistered the Portworx CSI driver immediately after initial registration. As a result, Portworx CSI volumes became stuck in the Pending state.

Resolution: This race condition and accidental deregistration no longer occurs. This change is applicable to Portworx versions 2.13 and above that have enabled CSI.
PWX-34360The csi-node-driver-registrar container ran inside the portworx pods. During a Kubernetes upgrade, when a node is cordoned, the portworx pod is drained. This deregistered the Portworx CSI driver as the container it was in was removed. Any application pod on that node that had not drained until this point became stuck in the Terminating state.

Resolution: The stability and availability of the csi driver during upgrades and maintenance processes is now improved, and the csi-node-driver-registrar container is no longer affected by node draining. This change is applicable to Portworx versions 2.13 and above that have enabled CSI.
PWX-35282Previously, when a Kubernetes node was cordoned, the Portworx pod was automatically deleted but would restart after a brief five-second interval. This behavior was intended to allow applications using Portworx volumes sufficient time to drain properly.

Resolution: In this release, when a node is cordoned prior to a drain, the Portworx pod will now be deleted and will not restart for the next five minutes, instead of the previous five seconds. This change is implemented following the movement of the csi-node-driver-registrar to the portworx-api daemonset, rendering the portworx pod unnecessary for this duration. This revised approach ensures more stability and less frequent disruptions in Portworx volumes management during node maintenance.

23.10.2​

January 23, 2024

note

Portworx Enterprise installation or upgrade is currently not supported on OCP 4.14 and above versions.

Fixes​

Portworx by PureStorage has fixed the following issues in this release:

Issue NumberIssue Description
PWX-35255The Portworx Operator did not create Autopilot Rule Objects (AROs) for Autopilot rules on clusters with KubeVirt VMs.

Resolution: The operator now successfully enables the creation of AROs on clusters with KubeVirt VMs.
PWX-35206Attempting to create a PX-Storev2 cluster with an io2 volume, whether specifying the IOPS or not, failed.

Resolution: The operator now successfully creates PX-Storev2 clusters without any failures.
PWX-35203The operator ran pre-flight checks on undesired environments.

Resolution: The Operator now only runs pre-flight checks on EKS clusters running Portworx 3.0.0 or above.
PWX-35016Prometheus alerts PXKvdbNodeViewUnhealthy and PXKvdbClusterViewUnhealthy used an incorrect metric label {$labels.node_id} to identify Portworx nodes.

Resolution: These alerts now correctly use the appropriate node label to display the Portworx node name in the alert.
PWX-35004Fresh installations overwrote explicitly set Stork and Autopilot images in the StorageCluster with an empty string, requiring users to update the StorageCluster again to correct this.

Resolution: The operator now honors user-specified images and updates the StorageCluster status accurately.
PWX-32111Pre-flight checks failed during fresh installations with PX-Security enabled on EKS clusters.

Resolution: Pre-flight checks now successfully complete in this environment.

23.10.1​

November 22, 2023

Fixes​

Issue NumberIssue Description
PWX-35067In certain scenarios, the Portworx Operator incorrectly auto-detected the cloud provider when using the cloud drive feature. This issue occurred when users left the spec.cloudStorage.provider field blank while operating on any of the following environments:
  • Bare metal
  • RKE2
  • vSphere

User impact: This issue may have prevented Portworx from starting successfully when it was freshly installed or updated.

Resolution: Portworx no longer auto detects cloud providers when the spec.cloudStorage.provider field in the StorageCluster spec is left blank.

23.10.0​

November 7, 2023

caution

In certain scenarios, the Portworx Operator incorrectly auto-detects the cloud provider when using the cloud drive feature. This issue occurs when you leave the spec.cloudStorage.provider field blank while running Portworx on any of the following environments:

  • Bare metal
  • RKE2
  • vSphere

If you encounter this issue, Portworx fails to start successfully when it is freshly installed or upgraded.

As a workaround, please update the spec.cloudStorage.provider field in your StorageCluster spec to the correct value for your deployment. Correct values are:

  • pure
  • vsphere
  • aws
  • azure
  • gce
  • ibm
  • oracle
  • csi

Improvements​

Improvement NumberImprovement Description
PWX-32147The Portworx Operator now supports the OpenShift dynamic console plugin for air-gapped clusters.
PWX-31732You can now list all pods deployed by the operator using the filter: operator.libopenstorage.org/managed-by=portworx.
PWX-29299You can now install Portworx on Oracle Kubernetes Engine (OKE) clusters using operator Helm charts.
PWX-25748Install Grafana and the associated Portworx dashboards by enabling spec.monitoring.grafana.enabled.

Fixes​

Issue NumberIssue Description
PWX-34239In GKE environments, the Portworx Operator encountered PVC provisioning issues due to a missing GKE installation chart file in Helm.

Resolution: The portworx.io/is-gke: "true" annotation has been added to the elasticsearch-sc StorageClass spec to resolve this issue.
PWX-33831A cordoned node could leave the Portworx kvdb-api pod in a Failed state, potentially interfering with the node's upgrade or maintenance processes.

Resolution: The operator now explicitly checks for kvdb-api pods in a Failed state and cleans them up.
PWX-31025Migrating from a Portworx DaemonSet deployment to an operator deployment sometimes resulted in the system being stuck in an Initializing state, despite the successful completion of the migration.

Resolution: The migration process from DaemonSet to Operator deployment has been improved to properly transition states.
PWX-30455While the StorageCluster spec allows adding custom container mounts, it was previously not possible to use the same mounts as the default Portworx installation directories.

Resolution: Custom mounts in the StorageCluster spec can now override the default Portworx directories. For instance, directories like /opt/pwx or /var/cores can be changed to different directories on a partition with more disk space.

Known issues (Errata)​

Issue NumberIssue Description
PWX-32111PX-Security is not currently supported on PX-Store V2, and the Operator pre-check will not proceed for this combination.

23.7.0​

September 12, 2023

Notes​

  • Enhanced Security: This release addresses security vulnerabilities to enhance overall security.

  • Tech Preview: The OCP Dynamic Plugin for air-gapped installs is currently in tech preview.

  • The Portworx Operator fully supports generic HTTP/HTTPS proxies. However, there is limited support for HTTPS proxies using SSL inspection, such as Next-Generation Firewalls that re-encrypt SSL traffic. To accommodate HTTPS proxy with SSL:

    • Portworx by PureStorage recommends configuring the Portworx StorageCluster, Portworx Operator, and License Server similarly to air-gapped environments.
    • For the Portworx StorageCluster, you can configure the proxy's self-signed Certification Authority (CA) certificate.
    note

    Telemetry connection to Pure1 with the Next-Generation Firewall is not supported.

Improvements​

Improvement NumberImprovement Description
PWX-32188The Portworx Operator now includes portworx.io/tls-cipher-suites and portworx.io/tls-min-version configuration parameters for the portworx-pvc-controller. These parameters allow for specifying TLS cipher-suites preferences and setting the minimum TLS version, respectively.

Note: Due to a Golang limitation, selecting VersionTLS13 disables the customization of TLS cipher-suites.
PWX-32147The operator now enables OCP dynamic plugin installation for air-gapped clusters, including support for custom image registries. Additionally, clusters with PX-Security enabled also support OCP dynamic plugins.
PWX-32011The operator can now utilize the PX_HTTPS_PROXY environment variable to configure the Envoy proxy to use an internal URL to connect to the destination host.
PWX-30520The operator offers enhanced security for JWT package.
PWX-27765The StorageCluster now displays more defined and detailed phases of installation and upgrade, such as initializing, running, degraded, uninstalling, and others, in the condition list.

Fixes​

Issue NumberIssue Description
PWX-32145Previously, OCP dynamic plugin images were not included in the px-versions ConfigMap.

Resolution: OCP dynamic plugin images are now correctly listed in the ConfigMap.
PWX-31944In air-gapped environments, the csi-ext-pod pod experienced startup failures due to the default inclusion of the csi-health-monitor-controller container.

Resolution: The csi-health-monitor-controller container has been disabled to ensure uninterrupted startup of the csi-ext-pod.
PWX-31915Occasionally, a cordoned node could leave the Portworx kvdb-api pod in a Completed state, potentially disrupting the node's upgrade or maintenance processes.

Resolution: The Portworx Operator now proactively checks for kvdb-api pods in a Completed state and removes any terminated pods to prevent interference.
PWX-31842On the PKS platform, restarting Portworx pods and services led to excessive mounts, slowed IO operations, and, in some cases, caused the host to become unresponsive.

Resolution: Users should upgrade the Portworx Operator to version 23.7.x and reboot affected PKS nodes to resolve these issues.

23.5.1​

July 11, 2023

Fixes​

Issue NumberIssue Description
PWX-32051The port used for telemetry could be configured as an NFS port in certain distributions, leading to conflicts.

Resolution: The Portworx Operator now uses port 9029 for telemetry in Portworx 3.0.0 and later versions to avoid this issue.
PWX-32073The CSI provisioner was issuing multiple requests for PVC provisioning, causing system delays.

Resolution: The CSI provisioner timeout has been updated to mitigate these delays.
PWX-32197In proxied Envoy versions greater than 1.22, a necessary configuration section was missing, preventing telemetry pods from starting.

Resolution: The configuration section for Envoy running with an HTTP proxy has been added, ensuring telemetry pods start as expected.

23.5.0​

June 13, 2023

Notes​

note

The PodSecurityPolicy resource is deprecated from Kubernetes 1.21 and unsupported from 1.25.x. Hence, you need to use either Pod Security Admission or third-party admission-plugin or both to impose restrictions on pods.

New features​

Portworx by Pure Storage is proud to introduce the following new features:

  • OpenShift users on OCP 4.12 or newer and Portworx Enterprise 3.0.2 or newer versions can now enable the Console plugin option during Portworx Operator installation or upgrade to use the Portworx Cluster dashboard within the OpenShift UI to manage their Portworx cluster. This avoids switching and navigating between different management interfaces to perform Day 2 operations.
  • The Portworx Operator now supports loading installation images into multiple custom registries for seamless Portworx installation for all Kubernetes installation environments (air gapped and non-air gapped). You need to update the path of the custom registry for each of these components in the version manifest prior to installation. For more information, see Install Portworx on Kubernetes with a custom container registry.

Improvements​

Improvement NumberImprovement Description
PWX-26156If your Portworx runs on Kubernetes version 1.26 and higher, the Portworx Operator auto-enables CSI in StorageCluster for both fresh installation and upgrade to ease volume migration.
PWX-27920the operator enables batching in metrics collector to reduce memory usage on large scale clusters.

Fixes​

Issue NumberIssue Description
PWX-28650The Portworx Operator used to allow Autopilot to access all resources on the cluster without any restrictions.

Resolution: The operator now enables Autopilot to provide selective access for the required resources (actions), thus minimizing the RBAC permissions.
PWX-30386StorageCluster warnings were issued when the operator tried to delete a non-existent component.

Resolution: Warning alerts are no longer issues in such scenarios now.
PWX-30737Stork scheduler pods were not created and px-csi-ext pods were stuck in a pending state.

Resolution: Update Stork scheduler ClusterRole to use Portworx SCC to create a Stork scheduler pod.
PWX-30943Prometheus pod crashed due to out of memory issue with default memory and CPU.

Resolution: Users can now control and enforce memory usage on their actual cluster deployment and set the required limit by editing the StorageCluster spec.
PWX-31551The latest OpenShift installations have introduced more restrictive SELinux policies, as a result, non-privileged pods cannot access the csi.sock CSI interface file.

Resolution: All Portworx CSI pods now run as privileged pods.

Known issues (Errata)​

Issue NumberIssue Description
PD-2156On OpenShift clusters running with Dynatrace, Portworx might not start after upgrading OCP version from 4.11 to 4.12

Workaround: Delete the Portworx pod on the node.

23.4.1​

September 1, 2023

Fixes​

Issue NumberIssue Description
PWX-32073During PVC provisioning, CSI CreateVolume function used to constantly retry volume creation causing CSI provisioner to become unresponsive leading to delays.

Resolution: CSI provisioner timeout value is updated from 10 seconds to 5 minutes to provide adequate time for volume creation.

23.4.0​

May 03, 2023

Improvements​

Improvement NumberImprovement Description
PWX-27168A new annotation is added to the Portworx Operator to let users customize Prometheus alert rules without the operator rolling back the change. Add the following annotations in the Prometheus spec to customize the alerts:
metadata:
annotations:
operator.libopenstorage.org/reconcile: "0"
PWX-24897You can configure Prometheus with these new flags in your StorageCluster spec for monitoring Portworx:
  • storageCluster.spec.monitoring.prometheus.replicas
  • storageCluster.Spec.monitoring.prometheus.retention
  • storageCluster.spec.monitoring.prometheus.retentionSize
  • storageCluster.spec.monitoring.prometheus.storage
  • storageCluster.spec.monitoring.prometheus.volumes
  • storageCluster.spec.monitoring.prometheus.volumeMounts
For more information about these flags, refer Monitoring configuration.

Fixes​

Issue NumberIssue Description
PWX-29409In a cluster, if there was a zone with no nodes available for Portworx, Operator failed to pick a default value for the MaxStorageNodesPerZone parameter.

Resolution: Operator now ignores zone(s) with no nodes and utilizes other nodes to calculate MaxStorageNodesPerZone parameter value.
PWX-29398Operator triggered nil panic error if no nodes were available to install Portworx.

Resolution: User interface displays an appropriate error message instead of panicking when there are no nodes available for Portworx installation.

23.3.1​

March 29, 2023

Improvements​

Improvement NumberImprovement Description
PWX-30005Improvement for air-gapped clusters: the Portworx Operator now checks for Pure1 connectivity when enabled for the first time. If a telemetry cert has not yet been created and Portworx cannot reach Pure1, the operator disables telemetry.

23.3.0​

March 22, 2023

Notes​

note

You must upgrade to Operator 23.3.0 to avoid an ImagePullError after April 3rd due to changes in the Kubernetes registry path. Kubernetes is freezing gcr.k8s.io and moving to the registry.k8s.io repository on 3rd of April. For more information, see the Kubernetes blog.

New features​

Portworx by Pure Storage is proud to introduce the following new features:

  • telemetry is now enabled by default.

    • On fresh installations, all clusters will have telemetry enabled when you generate a spec from PX-Central.
    • When you upgrade the Portworx Operator to version 23.3.0, telemetry will be enabled by default unless you disable telemetry in the StorageCluster spec, or when the PX_HTTPS_PROXY variable is configured.
    note

    For air-gapped clusters, you must disable telemetry explicitly during spec generation. To learn how to disable telemetry, see the air-gapped installation section. If you do not disable telemetry, telemetry pods will remain in the init state as Portworx fails to reach the Pure1 telemetry endpoint. This does not impact Portworx pods.

  • The StorageCluster spec for configuring Prometheus now contains the following new fields:

    • spec.Monitoring.Prometheus.Resources: Provides the ability to configure Prometheus resource usage, such as memory and CPU usage. If the resources field is not configured, default limits will be set to CPU 1, memory 800M, and ephemeral storage 5G.
    • spec.Monitoring.Prometheus.securityContext.runAsNonRootin: Provides the ability to configure the Prometheus service type, and the default value is set to true.

  • Added a new environment variable KUBELET_DIR. This variable can be used to specify a custom kubelet directory path.

  • Added an annotation portworx.io/scc-priority to the StorageCluster spec for configuring the priority of Portworx security context constraints (SCC).

Improvements​

Improvement NumberImprovement Description
PWX-28147When upgrading to Operator version 23.3.0, all CSI sidecar images will be updated to the latest versions.
PWX-28077Operator will now update Prometheus and Alertmanager CRDs.

Fixes​

Issue NumberIssue Description
PWX-28343During Portworx Operator upgrade, the old telemetry registration pod were not being deleted.

Resolution: Changed the update deployment strategy of px-telemetry-registration to Recreate. Now the old pods will be deleted before the new ones are created.
PWX-29531The prometheus-px-prometheus pods were not being created in OpenShift due to failed SCC validation.

Resolution: The prometheus-px-prometheus pods are now being created in OpenShift environments.
PWX-29565Upgrading OpenShift from version 4.11.x to 4.12.3 was failing for the Portworx cluster.

Resolution: Changed Portworx SCC default priority to nil.
PWX-28101If the kubelet path was not set to the default path, the CSI driver would fail to start, and the PVC could not be provisioned.

Resolution: Now the KUBELET_DIR environment variable can be used to specify a custom path for the CSI driver.

1.10.5​

March 07, 2023

Updates​

  • Added the newspec.updatestrategy.rollingupdate.minreadyseconds flag. During rolling updates, this flag will wait for all pods to be ready for at least minReadySeconds before updating the next batch of pods, where the size of the pod batch is specified through the spec.updateStrategy.rollingUpdate.maxUnavailable flag.

1.10.4​

February 22, 2023

Updates​

  • Added a new annotation portworx.io/is-oke=true to the StorageCluster spec to support Portworx deployment on the Oracle Container Engine for Kubernetes (OKE) cluster.

Fixes​

  • Fixed a bug where the Portworx PVC controller leader election resources conflicted with the resources used by the Kubernetes controller manager.
  • Fixed the Anthos Telemetry installation failure. Operator now allows two sidecar containers to run on the same node.

1.10.3​

January 27, 2023

Fixes​

  • An issue with a missing node name in the Portworx pod template in Operator version 1.10.2 sometimes scheduled the Portworx pod on a random node. This node name is no longer missing, and Portworx is now scheduled correctly.

1.10.2​

January 24, 2023

Updates​

  • Stork now uses KubeSchedulerConfiguration for Kubernetes version 1.23 or newer, so that pods are evenly distributed across all nodes in your cluster.

1.10.1​

Dec 5, 2022

Updates​

  • Added support for Kubernetes version 1.25, which includes:
    • Removed PodSecurityPolicy when deploying Portworx with Operator.
    • Upgraded the API version of PodDisruptionBudget from policy/v1beta1 to policy/v1
  • Added a UI option in the spec generator to configure Kubernetes version when you choose to deploy Portworx version 2.12.
  • The Portworx Operator is now deployed without verbose log by default. To enable it, add the --verbose argument to the operator deployment.
  • For CSI deployment, the px-csi-ext pods now set Stork as a scheduler in the px-csi-ext deployment spec.
  • The operator now chooses maxStorageNodesPerZone’s default value to efficiently manage the number of storage nodes in a cluster. For more details, see Manage the number of storage nodes.

1.10.0​

Oct 24, 2022

Notes​

caution

To enable telemetry for DaemonSet-based Portworx installations, you must migrate to an Operator-based installation, then upgrade to Portworx version 2.12 before enabling Pure1 integration. For more details, see this document.

Updates​

  • Pure1 integration has been re-architected to be more robust and use less memory. It is supported on Portworx version 2.12 clusters deployed with Operator version 1.10.
  • To reduce memory usage, added a new argument disable-cache-for to disable Kubernetes objects from controller runtime cache. For example,--disable-cache-for="Event,ConfigMap,Pod,PersistentVolume,PersistentVolumeClaim".
  • Operator now blocks Portworx installation if Portworx is uninstalled without a wipe and then reinstalled with a different name.
  • For a new installation, Operator now sets the max number of storage nodes per zone, so that the 3 storage nodes in the entire cluster are uniformly spread across zones.

Fixes​

  • Fixed a bug where DaemonSet migration was failing if the Portworx cluster ID was too long.

1.9.1​

Sep 8, 2022

Updates​

  • Added support for Kubernetes version 1.24:
    • Added docker.io prefix for component images deployed by Operator.
    • To determine Kubernetes master nodes, Operator now uses the control-plane node role instead of master.

Fixes​

  • In Operator 1.9.0, when you enabled the CSI snapshot controller explicitly in the StorageCluster, the csi-snapshot-controller sidecar containers might have been removed during an upgrade or restart operation. This issue is fixed in Operator 1.9.1.

1.9.0​

Aug 1, 2022

Updates​

  • Daemonset to Operator migration is now Generally Available. This includes the following features:
  • The ability to perform a dry run of the migration
  • Migration for generic helm chart from Daemonset to the Portworx Operator
  • Support for the OnDelete migration strategy
  • Support for various configurations such as external KVDB, custom volumes, environment variables, service type, and annotations
  • You can now use the generic helm chart to install Portworx with the Portworx Operator. Note: Only AWS EKS has been validated for cloud deployments.
  • Support for enabling pprof in order to get Portworx Operator container profiles for memory, CPU, and so on.
  • The operator now creates example CSI storage classes.
  • The operator now enables the CSI snapshot controller by default on Kubernetes 1.17 and newer.

Fixes​

  • Fixed an issue where KVDB pods were repeatedly created when a pod was in the evicted or outOfPods status.

Known issues (Errata)​

  • When you upgrade Operator to version 1.9.0, the snapshot controller containers are removed from px-csi-ext deployment when the installSnapshotController flag is set to true explicitly in the StorageCluster spec.
    Workaround: To fix this issue, either restart Operator or upgrade to a newer version.

1.8.1​

June 22, 2022

Updates​

  • Added support for Operator to run on IPv6 environment.
  • You can now enable CSI topology feature by setting the .Spec.CSI.Topology.Enabled flag to true in the StorageCluster CRD, the default value is false. The feature is only supported on FlashArray direct access volumes.
  • Operator now uses custom SecurityContextConstraints portworx instead of privileged on OpenShift.
  • You can now add custom annotations to any service created by Operator.
  • You can now configure ServiceType on any service created by Operator.

Fixes​

  • Fixed pod recreation race condition during OCP upgrade by introducing exponential back-off to pod recreation when the operator.libopenstorage.org/cordoned-restart-delay-secs annotation is not set.
  • Fixed the incorrect CSI provisioner arguments when custom image registry path contains ":".

1.8.0​

Apr 14, 2022

Updates​

  • Daemonset to operator migration is in Beta release.
  • Added support for passing custom labels to Portworx API service from StorageCluster.
  • Operator now enables the Autopilot component to communicate securely using tokens when PX-Security is enabled in the Portworx cluster.
  • Added field preserveFullCustomImageRegistry in StorageCluster spec to preserve full image path when using custom image registry.
  • Operator now retrieves the version manifest through proxy if PX_HTTP_PROXY is configured.
  • Stork, Stork scheduler, CSI, and PVC controller pods are now deployed with topologySpreadConstraints to distribute pod replicas across Kubernetes failure domains.
  • Added support for installing health monitoring sidecars from StorageCluster.
  • Added support for installing snapshot controller and CRD from StorageCluster.
  • The feature gate for CSI is now deprecated and replaced by setting spec.csi.enabled in StorageCluster.
  • Added support to enable hostPID to Portworx pods using the annotation portworx.io/host-pid="true" in StorageCluster.
  • Operator now sets fsGroupPolicy in the CSIDriver object to File. Previously it was not set explicitly, and the default value was ReadWriteOnceWithFsType.
  • Added skip-resource annotation to PX-Security Kubernetes secrets to skip backing them to the cloud.
  • Operator now sets the dnsPolicy of Portworx pod to ClusterFirstWithHostNet by default.
  • When using Cloud Storage, Operator validates that the node groups in StorageCluster use only one common label selector key across all node groups. It also validates that the value matches spec.cloudStorage.nodePoolLabel if a is present. If the value is not present, it automatically populates it with the value of the common label selector.

Fixes​

  • Fixed Pod Disruption Budget issue blocking Openshift upgrade on Metro DR setup.
  • Fixed Stork scheduler's pod anti-affinity by adding the label name: stork-scheduler to Stork scheduler deployments.
  • When a node level spec specifies a cloud storage configuration, we no longer set the cluster level default storage configuration. Before this fix, the node level cloud storage configuration would be overwritten.