Monitor PX-CSI with Prometheus

PX-CSI integrates with Prometheus to provide built-in metrics monitoring. This integration enables you to observe the health and performance of PX-CSI components across your cluster.

Monitoring is enabled by default. PX-CSI deploys Prometheus components into the portworx namespace, where Prometheus collects metrics from the PX-CSI controller and node plugin pods. You can view these metrics in the Prometheus UI or integrate them with tools such as Alertmanager and Grafana.

PX-CSI includes the following components as part of its monitoring stack:

• The Portworx Operator deploys the Prometheus Operator, which manages a custom resource defining the Prometheus configuration and deploys the Prometheus stack.

When monitoring is active, PX-CSI deploys the following resources in the portworx namespace:

• The Prometheus custom resource named px-prometheus, which defines retention periods, scrape intervals, and other Prometheus settings.
• A Prometheus instance, deployed as a StatefulSet, which collects and serves metrics.

PX-CSI creates and configures these components automatically. You do not need to deploy or manage them manually.

Monitoring on OpenShift Container Platform

PX-CSI uses the Prometheus instance that OpenShift deploys for monitoring, rather than deploying px-prometheus. On OpenShift, set the spec.monitoring.prometheus.enabled field to false and the spec.monitoring.prometheus.exportMetrics field to true in your StorageCluster.

Enable monitoring

Monitoring is enabled by default. If it is disabled during installation, you can enable it by editing the StorageCluster specification.

OpenShift Container Platform

StorageCluster

spec:
  monitoring:
    prometheus:
      enabled: false       # Do not deploy PX Prometheus. Use OpenShift’s Prometheus instead.
      exportMetrics: true  # Export metrics to OpenShift Prometheus

Other Kubernetes platforms

StorageCluster

spec:
  monitoring:
    prometheus:
      enabled: true        # Deploy PX-CSI Prometheus stack
      exportMetrics: true  # Export metrics via ServiceMonitors

note

The enabled field determines if PX-CSI deploys its own Prometheus Operator and Prometheus instance. The exportMetrics field controls whether ServiceMonitor resources are created to allow Prometheus to scrape metrics from PX-CSI components.

Disable monitoring

To disable monitoring, update the StorageCluster specification:

StorageCluster

spec:
  monitoring:
    prometheus:
      enabled: false
      exportMetrics: false

Verify monitoring

note

These steps are not applicable on OpenShift Container Platform, because PX-CSI uses the Prometheus instance deployed by OpenShift.

To confirm that monitoring is active and running as expected:

1. Run the following command to check for Prometheus pods in the portworx namespace:

   kubectl -n portworx get pods | grep prometheus

   The output should include pod names similar to:

   prometheus-px-prometheus-0                2/2     Running   0              23h
   px-prometheus-operator-764bb9c6cb-9qgvd   1/1     Running   0              23h

2. Run the following command to forward Prometheus to port 9090:

   kubectl -n portworx port-forward prometheus-px-prometheus-0 9090:9090

3. Open http://localhost:9090/targets to view the Prometheus targets.

   Verify that targets such as px-pure-csi-controller and px-pure-csi-node are listed and have a status of UP.

Prometheus metrics

PX-CSI metrics contain information about PersistentVolumeClaim (PVC) usage, volume lifecycle operations, API requests to external endpoints, and CSI call latencies.

API latency metrics

Metric	Type	Description
px_csi_create_volume_latency_ms	Histogram	Latency histogram for the CreateVolume API
px_csi_delete_volume_latency_ms	Histogram	Latency histogram for the DeleteVolume API
px_csi_ctrlpublishvolume_latency_ms	Histogram	Latency histogram for the ControllerPublishVolume API
px_csi_ctrlunpublishvolume_latency_ms	Histogram	Latency histogram for the ControllerUnpublishVolume API
px_csi_nodestagevolume_latency_ms	Histogram	Latency histogram for the NodeStageVolume API
px_csi_nodeunstagevolume_latency_ms	Histogram	Latency histogram for the NodeUnstageVolume API
px_csi_nodepublishvolume_latency_ms	Histogram	Latency histogram for the NodePublishVolume API
px_csi_nodeunpublishvolume_latency_ms	Histogram	Latency histogram for the NodeUnpublishVolume API

Volume attachment metric

Metric	Type	Description
px_csi_attachments_per_node	Gauge	Number of volume attachments per node

Volume usage metrics

Starting with PX-CSI 26.1.0, PX-CSI exposes volume usage metrics for filesystem volumes.

Metric	Type	Description	Labels
px_volume_fs_usage_bytes	Gauge	Current filesystem used bytes in the volume	node, volumeid, volumename, pvc
px_volume_capacity_bytes	Gauge	Total capacity of the volume in bytes	node, volumeid, volumename, pvc

FlashArray and FlashBlade API request metrics

Metric	Type	Description
px_csi_fafb_all_apis_requests_total	Counter	Total API requests to all configured FA/FB endpoints
px_csi_fafb_apis_volumes_requests_total	Counter	API requests to the /volumes endpoint
px_csi_fafb_apis_array_requests_total	Counter	API requests to the /arrays endpoint
px_csi_fafb_apis_volumesnapshots_requests_total	Counter	API requests to the /volume-snapshots endpoint
px_csi_fafb_apis_hosts_requests_total	Counter	API requests to the /hosts endpoint
px_csi_fafb_apis_controllers_requests_total	Counter	API requests to the /controllers endpoint
px_csi_fafb_apis_ports_requests_total	Counter	API requests to the /ports endpoint
px_csi_fafb_apis_alerts_requests_total	Counter	API requests to the /alerts endpoint
px_csi_fafb_apis_connections_requests_total	Counter	API requests to the /connections endpoint
px_csi_fafb_apis_login_requests_total	Counter	API requests to the /login endpoint
px_csi_fafb_apis_version_requests_total	Counter	API requests to the /api_version endpoint

Kubelet volume metrics

Starting with PX-CSI 26.1.0, kubelet exposes standard Kubernetes volume metrics for PX-CSI volumes.

Metric	Type	Description
kubelet_volume_stats_capacity_bytes	Gauge	Total capacity of the volume in bytes
kubelet_volume_stats_used_bytes	Gauge	Number of used bytes in the volume
kubelet_volume_stats_available_bytes	Gauge	Number of available bytes remaining in the volume
kubelet_volume_stats_inodes	Gauge	Total number of inodes in the volume (filesystem volumes only)
kubelet_volume_stats_inodes_used	Gauge	Number of used inodes in the volume (filesystem volumes only)
kubelet_volume_stats_inodes_free	Gauge	Number of free inodes in the volume (filesystem volumes only)