Portworx Metrics for monitoring
Alert metrics
Alert metrics use variable labels depending on the resource type. Each metric has a label named after its resource type (such as node, volume, drive, or cluster) that contains the resource ID, plus a clusterid label.
| Name | Labels | Description |
|---|---|---|
| px_alerts_callhomefailure | node, clusterid | Increments when a call-home failure alert is raised; decrements when cleared |
| px_alerts_nodestatechange | node, clusterid | Increments when a node state-change alert is raised; decrements when cleared |
| px_alerts_nodestartsuccess | node, clusterid | Increments when a node successfully starts; decrements when cleared |
Backup metrics
Per-backup metrics include volume and volumename labels that identify the backed-up volume. Node-level transfer metrics (px_backup_stats_node_*) do not include volume labels.
| Name | Labels | Description |
|---|---|---|
| px_backup_stats_backup_status | cluster, clusterUUID, nodeID, node, volume, volumename | Status for this backup (0=InProgress,1=Done,2=Failed) |
| px_backup_stats_backup_size | cluster, clusterUUID, nodeID, node, volume, volumename | Size in bytes for this backup |
| px_backup_stats_backup_duration_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Duration in seconds for this backup |
| px_backup_stats_backup_uploaded_bytes_mbs | cluster, clusterUUID, nodeID, node, volume, volumename | Delta bytes uploaded in MB/s from last interval for this backup |
| px_backup_stats_download_size_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Size of downloaded bytes during backup/restore from cloud |
| px_backup_stats_upload_size_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Size of uploaded bytes during backup/restore from cloud |
| px_backup_stats_get_apis_invoked | cluster, clusterUUID, nodeID, node, volume, volumename | Number of times GET API was invoked |
| px_backup_stats_put_apis_invoked | cluster, clusterUUID, nodeID, node, volume, volumename | Number of times PUT API was invoked |
| px_backup_stats_read_time_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Time spent in read operations during cloudsnap backup |
| px_backup_stats_write_time_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Time spent in write operations during cloudsnap restore |
| px_backup_stats_upload_time_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Time spent in upload operations during cloudsnap backup |
| px_backup_stats_download_time_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Time spent in download operations during cloudsnap restore |
| px_backup_stats_compress_time_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Time spent in compression during cloudsnap backup |
| px_backup_stats_encrypt_time_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Time spent in encryption during cloudsnap backup |
| px_backup_stats_node_upload_bytes_total | cluster, clusterUUID, nodeID, node | Total bytes uploaded from this node across all cloudsnaps |
| px_backup_stats_node_download_bytes_total | cluster, clusterUUID, nodeID, node | Total bytes downloaded to this node across all cloudsnaps |
| px_backup_stats_node_upload_bytes_interval | cluster, clusterUUID, nodeID, node | Bytes uploaded from this node in the current interval |
| px_backup_stats_node_download_bytes_interval | cluster, clusterUUID, nodeID, node | Bytes downloaded to this node in the current interval |
| px_backup_stats_node_upload_rate_bytes_per_sec | cluster, clusterUUID, nodeID, node | Current upload rate in bytes per second for this node |
| px_backup_stats_node_download_rate_bytes_per_sec | cluster, clusterUUID, nodeID, node | Current download rate in bytes per second for this node |
Cluster metrics
| Name | Labels | Description |
|---|---|---|
| px_cluster_cpu_percent | cluster, clusterUUID, nodeID, node | Percentage of CPU used |
| px_cluster_memory_utilized_percent | cluster, clusterUUID, nodeID, node | Percentage of memory utilization |
| px_cluster_disk_total_bytes | cluster, clusterUUID, nodeID, node | Total storage space in bytes for this node |
| px_cluster_disk_available_bytes | cluster, clusterUUID, nodeID, node | Available storage space in bytes for this node |
| px_cluster_disk_utilized_bytes | cluster, clusterUUID, nodeID, node | Utilized storage space in bytes for this node |
| px_cluster_pendingio | cluster, clusterUUID, nodeID, node | Number of read and write operations currently in progress for this node |
| px_cluster_trashcan_volumes | cluster, clusterUUID, nodeID, node | Number of volumes currently in the trashcan |
Cluster status metrics
| Name | Labels | Description |
|---|---|---|
| px_cluster_status_cluster_size | cluster, clusterUUID, nodeID, node | Node count for your Portworx cluster. Deprecated. |
| px_cluster_status_size | cluster, clusterUUID, nodeID, node | Node count for your Portworx cluster |
| px_cluster_status_cluster_quorum | cluster, clusterUUID, nodeID, node | Indicates if the cluster is in quorum. Deprecated. |
| px_cluster_status_quorum | cluster, clusterUUID, nodeID, node | Indicates if the cluster is in quorum |
| px_cluster_status_nodes_online | cluster, clusterUUID, nodeID, node | Number of online nodes in the cluster (includes storage and storageless) |
| px_cluster_status_nodes_offline | cluster, clusterUUID, nodeID, node | Number of offline nodes in the cluster (includes storage and storageless) |
| px_cluster_status_nodes_storage_down | cluster, clusterUUID, nodeID, node | Number of nodes where the storage is full or down |
| px_cluster_status_storage_nodes_online | cluster, clusterUUID, nodeID, node | Number of storage nodes that are online |
| px_cluster_status_storage_nodes_offline | cluster, clusterUUID, nodeID, node | Number of storage nodes that are offline |
| px_cluster_status_storage_nodes_decommissioned | cluster, clusterUUID, nodeID, node | Number of storage nodes that are decommissioned |
Device delete metrics
| Name | Labels | Description |
|---|---|---|
| px_device_delete_delete_after_discard_enabled | cluster, clusterUUID, nodeID, node, poolid | Current configuration value indicating if NBDD is enabled |
| px_device_delete_discard_bytes_total | cluster, clusterUUID, nodeID, node, poolid | Total bytes discarded across all device deletes |
| px_device_delete_discard_duration_msec_total | cluster, clusterUUID, nodeID, node, poolid | Total time spent in discard operations in milliseconds across all device deletes |
| px_device_delete_discard_errors_total | cluster, clusterUUID, nodeID, node, poolid | Total number of discard errors encountered across all device deletes |
| px_device_delete_discard_operations_total | cluster, clusterUUID, nodeID, node, poolid | Total number of discard operations performed across all device deletes |
| px_device_delete_discard_size_bytes | cluster, clusterUUID, nodeID, node, poolid | Current configuration value for discard size in bytes |
| px_device_delete_operations_completed_total | cluster, clusterUUID, nodeID, node, poolid | Total number of delete operations completed |
| px_device_delete_operations_inprogress | cluster, clusterUUID, nodeID, node, poolid | Current number of delete operations in progress |
| px_device_delete_operations_queued | cluster, clusterUUID, nodeID, node, poolid | Current number of delete operations queued |
| px_device_delete_release_bytes_total | cluster, clusterUUID, nodeID, node, poolid | Total size of devices processed for storage release in bytes |
| px_device_delete_release_duration_msec_total | cluster, clusterUUID, nodeID, node, poolid | Total time spent in storage release operations in milliseconds |
| px_device_delete_release_operations_total | cluster, clusterUUID, nodeID, node, poolid | Total number of devices processed for storage release |
Disk metrics
| Name | Labels | Description |
|---|---|---|
| px_disk_stats_used_bytes | cluster, clusterUUID, nodeID, node, disk, pool | Total storage in bytes for this disk |
| px_disk_stats_interval_seconds | cluster, clusterUUID, nodeID, node, disk, pool | interval_seconds |
| px_disk_stats_io_seconds | cluster, clusterUUID, nodeID, node, disk, pool | Time spent doing IO in seconds for this disk |
| px_disk_stats_progress_io | cluster, clusterUUID, nodeID, node, disk, pool | IO's currently in progress for this disk |
| px_disk_stats_disk_read_bytes | cluster, clusterUUID, nodeID, node, disk, pool | Total bytes read for this disk. Deprecated. |
| px_disk_stats_read_bytes | cluster, clusterUUID, nodeID, node, disk, pool | Total bytes read for this disk |
| px_disk_stats_write_bytes_seconds | cluster, clusterUUID, nodeID, node, disk, pool | Total written bytes for this disk. Deprecated. |
| px_disk_stats_written_bytes | cluster, clusterUUID, nodeID, node, disk, pool | Total written bytes for this disk |
| px_disk_stats_read_seconds | cluster, clusterUUID, nodeID, node, disk, pool | Total time spent reading in seconds for this disk |
| px_disk_stats_write_seconds | cluster, clusterUUID, nodeID, node, disk, pool | Total time spent writing in seconds for this disk |
| px_disk_stats_read_latency_seconds | cluster, clusterUUID, nodeID, node, disk, pool | Average time spent per read operation in seconds for this disk |
| px_disk_stats_write_latency | cluster, clusterUUID, nodeID, node, disk, pool | Average time spent per write operation in seconds for this disk. Deprecated. |
| px_disk_stats_write_latency_seconds | cluster, clusterUUID, nodeID, node, disk, pool | Average time spent per write operation in seconds |
| px_disk_stats_disk_num_reads | cluster, clusterUUID, nodeID, node, disk, pool | Total number of read operations completed successfully for this disk. Deprecated. |
| px_disk_stats_disk_num_writes | cluster, clusterUUID, nodeID, node, disk, pool | Total number of write operations completed successfully for this disk. Deprecated. |
| px_disk_stats_num_reads | cluster, clusterUUID, nodeID, node, disk, pool | Total number of read operations completed successfully for this disk |
| px_disk_stats_num_writes | cluster, clusterUUID, nodeID, node, disk, pool | Total number of write operations completed successfully for this disk |
| px_disk_stats_num_reads_total | cluster, clusterUUID, nodeID, node, disk, pool | Total number of read operations completed successfully for this disk |
| px_disk_stats_num_writes_total | cluster, clusterUUID, nodeID, node, disk, pool | Total number of write operations completed successfully for this disk |
| px_disk_stats_written_bytes_total | cluster, clusterUUID, nodeID, node, disk, pool | Total bytes written for this disk |
| px_disk_stats_read_bytes_total | cluster, clusterUUID, nodeID, node, disk, pool | Total bytes read for this disk |
| px_disk_stats_read_seconds_total | cluster, clusterUUID, nodeID, node, disk, pool | Total time spent reading in seconds for this disk |
| px_disk_stats_write_seconds_total | cluster, clusterUUID, nodeID, node, disk, pool | Total time spent writing in seconds for this disk |
FA/FB API metrics
| Name | Labels | Description |
|---|---|---|
| px_fafb_apis_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to all the configured FA/FB endpoints |
| px_fafb_apis_array_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /arrays endpoint |
| px_fafb_apis_login_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /login endpoint |
| px_fafb_apis_controllers_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /controllers endpoint |
| px_fafb_apis_alerts_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /alerts endpoint |
| px_fafb_apis_ports_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /ports endpoint |
| px_fafb_apis_volumes_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /volumes endpoint |
| px_fafb_apis_volumesnapshots_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /volume-snapshots endpoint |
| px_fafb_apis_hosts_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /hosts endpoint |
| px_fafb_apis_connections_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /connections endpoint |
| px_fafb_apis_version_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /api_version endpoint |
Kube Datastore metrics
Use the following Prometheus metrics to monitor Kube Datastore capacity and performance:
| Name | Labels | Description |
|---|---|---|
px_kube_datastore_total_bytes | kube_datastore | Total capacity, in bytes, across all storage pools in the Kube Datastore. |
px_kube_datastore_used_bytes | kube_datastore | Used capacity, in bytes, across all storage pools in the Kube Datastore. |
px_kube_datastore_available_bytes | kube_datastore | Available capacity, in bytes, across all storage pools in the Kube Datastore. |
px_kube_datastore_provisioned_bytes | kube_datastore | Provisioned volume capacity, in bytes, for the Kube Datastore. |
px_kube_datastore_used_percent | kube_datastore | Used capacity as a percentage of the total Kube Datastore capacity. |
px_kube_datastore_provisioned_percent | kube_datastore | Provisioned capacity as a percentage of the total Kube Datastore capacity. |
px_kube_datastore_write_iops | kube_datastore | Write IOPS for the Kube Datastore, calculated as a 5-minute rate. |
px_kube_datastore_write_throughput_bytes_per_second | kube_datastore | Write throughput, in bytes per second, for the Kube Datastore, calculated as a 5-minute rate. |
px_kube_datastore_write_latency_seconds | kube_datastore | Write latency, in seconds, for the Kube Datastore, calculated over a 5-minute rate. |
note
-
Kube Datastore metrics are generated from storage pool metrics by using Prometheus recording rules installed by the Portworx Operator. Prometheus is required to evaluate these rules. On OpenShift, additional configuration is required to enable the recording rules. For more information, see Enable Kube Datastore metrics.
-
Storage pool metrics include the
kube_datastorelabel, which identifies the Kube Datastore associated with each pool. -
Kube Datastore I/O metrics are available only for write operations because storage pool metrics for read operations are not available.
KVDB metrics
| Name | Labels | Description |
|---|---|---|
| px_kvdb_get_requests_total | cluster, clusterUUID, nodeID, node | Total number of get requests for a key |
| px_kvdb_snapshot_requests_total | cluster, clusterUUID, nodeID, node | Total number of kvdb snapshot requests for a list of prefixes |
| px_kvdb_put_requests_total | cluster, clusterUUID, nodeID, node | Total number of put requests for a key |
| px_kvdb_create_requests_total | cluster, clusterUUID, nodeID, node | Total number of create requests for a key |
| px_kvdb_update_requests_total | cluster, clusterUUID, nodeID, node | Total number of update requests for a key |
| px_kvdb_enumerate_requests_total | cluster, clusterUUID, nodeID, node | Total number of enumerate requests for a key |
| px_kvdb_delete_requests_total | cluster, clusterUUID, nodeID, node | Total number of delete requests for a key |
| px_kvdb_keys_requests_total | cluster, clusterUUID, nodeID, node | Total number of list key requests for a prefix |
| px_kvdb_cas_requests_total | cluster, clusterUUID, nodeID, node | Total number of compare and sets for a key |
| px_kvdb_cad_requests_total | cluster, clusterUUID, nodeID, node | Total number of compare and deletes for a key |
| px_kvdb_lock_requests_total | cluster, clusterUUID, nodeID, node | Total number of lock requests for a key |
| px_kvdb_unlock_requests_total | cluster, clusterUUID, nodeID, node | Total number of unlock requests for a key |
| px_kvdb_watchkey_requests_total | cluster, clusterUUID, nodeID, node | Total number of watch requests for a key from a node |
| px_kvdb_watchtree_requests_total | cluster, clusterUUID, nodeID, node | Total number of watch requests for a prefix from a node |
| px_kvdb_adduser_requests_total | cluster, clusterUUID, nodeID, node | Total number of add user requests |
| px_kvdb_removeuser_requests_total | cluster, clusterUUID, nodeID, node | Total number of remove user requests |
| px_kvdb_grantuser_access_requests_total | cluster, clusterUUID, nodeID, node | Total number of grant access requests for a user |
| px_kvdb_revokeuser_access_requests_total | cluster, clusterUUID, nodeID, node | Total number of revoke access requests for a user |
| px_kvdb_addmember_requests_total | cluster, clusterUUID, nodeID, node | Total number of add member requests from a node |
| px_kvdb_addlearner_requests_total | cluster, clusterUUID, nodeID, node | Total number of add learner requests from a node |
| px_kvdb_removemember_requests_total | cluster, clusterUUID, nodeID, node | Total number of remove member requests from a node |
| px_kvdb_updatemember_requests_total | cluster, clusterUUID, nodeID, node | Total number of update member requests from a node |
| px_kvdb_listmembers_requests_total | cluster, clusterUUID, nodeID, node | Total number of list member requests from a node |
| px_kvdb_setendpoints_requests_total | cluster, clusterUUID, nodeID, node | Total number of set endpoint requests from a node |
| px_kvdb_getendpoints_requests_total | cluster, clusterUUID, nodeID, node | Total number of get endpoint requests from a node |
| px_kvdb_defragment_requests_total | cluster, clusterUUID, nodeID, node | Total number of defragment requests from a node |
| px_kvdb_watch_latency_seconds | cluster, clusterUUID, nodeID, node | Time taken in seconds between a kvdb put and a corresponding watch update |
KVDB health metrics
| Name | Labels | Description |
|---|---|---|
| px_kvdb_health_state_node_view | cluster, clusterUUID, nodeID, node | This node's kvdb state (1 = healthy, 2 = not healthy) |
| px_kvdb_health_state_cluster_view | cluster, clusterUUID, nodeID, node | This node's view of other node's kvdb state (1 = healthy, 2 = not healthy) |
License metrics
| Name | Labels | Description |
|---|---|---|
| px_license_expiry | cluster, clusterUUID, nodeID, node | License expiry in days. A negative value indicates the license has expired. |
| px_license_next_refresh | cluster, clusterUUID, nodeID, node | Next license refresh timestamp in minutes. A negative value indicates that the license will not be refreshed. |
| px_license_type | cluster, clusterUUID, nodeID, node | Indicates the type of license (0=PX-Developer, 1=Legacy Enterprise, 2=Trial, 3=PX-Enterprise, 4=PX-Enterprise floating, 5=IBM Cloud Pak for Data, 6=CISCO OEM, 7=PX-Essential, 8=Portworx CSI for FA/FB, 9=INVALID LICENSE) |
Network I/O metrics
| Name | Labels | Description |
|---|---|---|
| px_network_io_bytessent | cluster, clusterUUID, nodeID, node | Number of bytes sent during this interval |
| px_network_io_received_bytes | cluster, clusterUUID, nodeID, node | Number of bytes received during this interval |
| px_network_io_sent_bytes_total | cluster, clusterUUID, nodeID, node | Total number of bytes sent |
| px_network_io_received_bytes_total | cluster, clusterUUID, nodeID, node | Total number of bytes received |
| px_network_io_reservation_waits | cluster, clusterUUID, nodeID, node, nodeid | Total number of SM waits for reservation |
| px_network_io_reservation_wake_ups | cluster, clusterUUID, nodeID, node, nodeid | Total number of SM wake ups after getting the reservation |
NFS pool metrics
| Name | Labels | Description |
|---|---|---|
| px_nfs_pool_stats_packets_arrived | cluster, clusterUUID, nodeID, node | Counts the number of NFS packets that arrived on the socket. |
| px_nfs_pool_stats_sockets_enqueued | cluster, clusterUUID, nodeID, node | Counts how many times an NFS transport is enqueued to wait for an nfsd thread to service it, i.e. no nfsd thread was considered available. |
| px_nfs_pool_stats_threads_woken | cluster, clusterUUID, nodeID, node | Counts how many times an idle nfsd thread is woken to try to receive some data from an NFS transport. |
| px_nfs_pool_stats_threads_timedout | cluster, clusterUUID, nodeID, node | Counts how many times an nfsd thread triggered an idle timeout, i.e. was not woken to handle any incoming network packets for some time. |
Node metrics
| Name | Labels | Description |
|---|---|---|
| px_node_stats_free_mem | cluster, clusterUUID, nodeID, node | Available memory in bytes |
| px_node_stats_total_mem | cluster, clusterUUID, nodeID, node | Total memory in bytes |
| px_node_stats_used_mem | cluster, clusterUUID, nodeID, node | Used memory in bytes |
| px_node_stats_cpu_usage | cluster, clusterUUID, nodeID, node | Percent of CPU consumption |
| px_node_stats_busy_threads_percent | cluster, clusterUUID, nodeID, node | Percentage of busy threads |
| px_node_stats_thread_pool_high_priority_queue_size | cluster, clusterUUID, nodeID, node | Thread Pool High Priority Queue Size |
| px_node_stats_thread_pool_long_running_busy_threads | cluster, clusterUUID, nodeID, node | Long Running Thread Pool Number Of Busy Threads |
| px_node_stats_thread_pool_long_running_queue_size | cluster, clusterUUID, nodeID, node | Long Running Thread Pool Pending Items In Queue |
| px_node_stats_thread_pool_long_running_max_threads | cluster, clusterUUID, nodeID, node | Long Running Thread Pool Count Of Max Threads |
| px_node_stats_writeback_list_num_dirty_pools | cluster, clusterUUID, nodeID, node | Writeback List Number of Dirty Pools |
| px_node_stats_writeback_list_num_pending_log_bufs | cluster, clusterUUID, nodeID, node | Writeback List Number of Pending Log Buffers |
| px_node_stats_writeback_list_low_cos_queue_size | cluster, clusterUUID, nodeID, node | Writeback List Low Cos Queue Size |
| px_node_stats_writeback_list_high_cos_queue_size | cluster, clusterUUID, nodeID, node | Writeback List High Cos Queue Size |
| px_node_stats_sm_run_queue_size | cluster, clusterUUID, nodeID, node | Statemachine Run Queue Size |
| px_node_stats_kaio_pending_queue_size | cluster, clusterUUID, nodeID, node | Kaio Pending Queue Size |
| px_node_stats_request_scheduler_queue_size | cluster, clusterUUID, nodeID, node | Request Scheduler Queue Size |
| px_node_stats_num_reads | cluster, clusterUUID, nodeID, node | Number of read operations in the last interval for this node |
| px_node_stats_read_bytes | cluster, clusterUUID, nodeID, node | Read bytes in the last interval for this node |
| px_node_stats_read_ms | cluster, clusterUUID, nodeID, node | Total time in milliseconds spent reading in the last interval for this node |
| px_node_stats_read_iops | cluster, clusterUUID, nodeID, node | Average number of completed read operations per second for this node |
| px_node_stats_readthroughput | cluster, clusterUUID, nodeID, node | Average number of bytes read per second for this node |
| px_node_stats_read_latency_seconds | cluster, clusterUUID, nodeID, node | Average time spent per read operation for this node |
| px_node_stats_num_writes | cluster, clusterUUID, nodeID, node | Number of write operations in the last interval for this node |
| px_node_stats_written_bytes | cluster, clusterUUID, nodeID, node | Bytes written since last interval for this node |
| px_node_stats_write_ms | cluster, clusterUUID, nodeID, node | Total time in milliseconds spent writing in the last interval for this node |
| px_node_stats_write_iops | cluster, clusterUUID, nodeID, node | Average number of completed write operations per second for this node |
| px_node_stats_writethroughput | cluster, clusterUUID, nodeID, node | Average number of bytes written per second for this node |
| px_node_stats_write_latency_seconds | cluster, clusterUUID, nodeID, node | Average time spent per write operation for this node |
| px_node_stats_num_discards | cluster, clusterUUID, nodeID, node | Number of discard operations in the last interval for this node |
| px_node_stats_discarded_bytes | cluster, clusterUUID, nodeID, node | Bytes discarded since last interval for this node |
| px_node_stats_discard_ms | cluster, clusterUUID, nodeID, node | Total time in milliseconds spent discarding in the last interval for this node |
| px_node_stats_discard_iops | cluster, clusterUUID, nodeID, node | Average number of completed discard operations per second for this node |
| px_node_stats_discardthroughput | cluster, clusterUUID, nodeID, node | Average number of bytes discarded per second for this node |
| px_node_stats_discard_latency_seconds | cluster, clusterUUID, nodeID, node | Average time spent per discard operation for this node |
| px_node_stats_num_flushes | cluster, clusterUUID, nodeID, node | Number of flush(sync) operations in the last interval for this node |
| px_node_stats_flushed_bytes | cluster, clusterUUID, nodeID, node | Number of flushed bytes since last interval for this node |
| px_node_stats_flushms | cluster, clusterUUID, nodeID, node | Latency for flush for this node |
| px_node_stats_flush_iops | cluster, clusterUUID, nodeID, node | Average number of completed flush operations per second for this node |
| px_node_stats_flushthroughput | cluster, clusterUUID, nodeID, node | Average number of bytes flushed per second for this node |
| px_node_stats_flush_latency_seconds | cluster, clusterUUID, nodeID, node | Average time spent per flush operation for this node |
| px_node_stats_num_ios | cluster, clusterUUID, nodeID, node | Number of IO operations in the last interval for this node |
| px_node_stats_io_bytes | cluster, clusterUUID, nodeID, node | IO bytes in the last interval for this node |
| px_node_stats_io_ms | cluster, clusterUUID, nodeID, node | Total time in milliseconds spent doing IO in the last interval for this node |
| px_node_stats_io_iops | cluster, clusterUUID, nodeID, node | Average number of completed IO operations per second for this node |
| px_node_stats_iothroughput | cluster, clusterUUID, nodeID, node | Average number of bytes IO per second for this node |
| px_node_stats_io_latency_seconds | cluster, clusterUUID, nodeID, node | Average time spent per IO operation for this node |
| px_node_stats_procfs_mem_available_bytes | cluster, clusterUUID, nodeID, node | Available memory in bytes |
| px_node_stats_procfs_mem_dirty_bytes | cluster, clusterUUID, nodeID, node | The total amount of memory waiting to be written back to the disk |
| px_node_stats_procfs_mem_writeback_bytes | cluster, clusterUUID, nodeID, node | The total amount of memory actively being written back to the disk |
| px_node_stats_procfs_mem_total_bytes | cluster, clusterUUID, nodeID, node | Total amount of usable RAM which is physical RAM minus a number of reserved bits and the kernel binary code |
| px_node_stats_procfs_mem_free_bytes | cluster, clusterUUID, nodeID, node | The amount of physical RAM, in bytes, left unused by the system. |
| px_node_stats_procfs_mem_buffers_bytes | cluster, clusterUUID, nodeID, node | The amount, in bytes, of temporary storage for raw disk blocks |
| px_node_stats_procfs_mem_cached_bytes | cluster, clusterUUID, nodeID, node | The amount of physical RAM, in bytes, used as cache memory |
| px_node_stats_procfs_mem_active_anon_bytes | cluster, clusterUUID, nodeID, node | The amount of anonymous and tmpfs/shmem memory, in bytes, that is in active use, or was in active use since the last time the system moved something to swap |
| px_node_stats_procfs_mem_inactive_anon_bytes | cluster, clusterUUID, nodeID, node | The amount of anonymous and tmpfs/shmem memory, in bytes, that is a candidate for eviction |
| px_node_stats_procfs_mem_active_file_bytes | cluster, clusterUUID, nodeID, node | The amount of file cache memory, in bytes, that is in active use, or was in active use since the last time the system reclaimed memory |
| px_node_stats_procfs_mem_inactive_file_bytes | cluster, clusterUUID, nodeID, node | The amount of file cache memory, in bytes, that is newly loaded from the disk, or is a candidate for reclaiming |
| px_node_stats_procfs_mem_unevictable_bytes | cluster, clusterUUID, nodeID, node | The amount of memory, in bytes, discovered by the pageout code, that is not evictable because it is locked into memory by user programs |
| px_node_stats_procfs_mem_mlocked_bytes | cluster, clusterUUID, nodeID, node | The total amount of memory, in bytes, that is not evictable because it is locked into memory by user programs |
| px_node_stats_procfs_mem_anon_pages_bytes | cluster, clusterUUID, nodeID, node | The total amount of memory, in bytes, used by pages that are not backed by files and are mapped into userspace page tables |
| px_node_stats_procfs_mem_mapped_bytes | cluster, clusterUUID, nodeID, node | The memory, in bytes, used for files that have been mmaped, such as libraries |
| px_node_stats_procfs_mem_sh_mem_bytes | cluster, clusterUUID, nodeID, node | The total amount of memory, in bytes, used by shared memory (shmem) and tmpfs |
| px_node_stats_num_skinnysnaps | cluster, clusterUUID, nodeID, node | The number of SkinnySnaps |
| px_node_stats_skinnysnaps_num_repls_skips | cluster, clusterUUID, nodeID, node | The number of target replicas skipped for a snapshot due to SkinnySnap |
| px_node_stats_skinnysnaps_num_repls_snapshots | cluster, clusterUUID, nodeID, node | The number of target replicas that underwent a snapshot operation due to SkinnySnap |
| px_node_stats_relaxed_reclaim_pending | cluster, clusterUUID, nodeID, node | The number of volumes in RelaxedReclaim pending queue |
| px_node_stats_relaxed_reclaim_skipped | cluster, clusterUUID, nodeID, node | The number of volume deletes that were skipped from being delayed even when RelaxedReclaim was enabled. |
| px_node_stats_relaxed_reclaim_deleted | cluster, clusterUUID, nodeID, node | The number of volume deletes done through RelaxedReclaim |
| px_node_stats_num_volumes | cluster, clusterUUID, nodeID, node | The number of volumes hosted on this node |
| px_node_stats_local_snapshot_count | cluster, clusterUUID, nodeID, node | Total number of local snapshots present on this node. If a snapshot has HA>1, it will be counted on each node where it is replicated. |
| px_node_stats_cloud_snapshot_count | cluster, clusterUUID, nodeID, node | Total number of active/in-progress cloud snapshots from this node (Active, Queued states). |
| px_node_stats_group_snapshot_count | cluster, clusterUUID, nodeID, node | Total number of unique group IDs on this node (counts distinct group snapshots including local snapshots and completed cloud backups). |
| px_node_stats_local_group_snapshot_count | cluster, clusterUUID, nodeID, node | Total number of unique local group snapshot IDs present on this node (excludes cascaded cloud-backup references). |
| px_node_stats_cloud_group_snapshot_count | cluster, clusterUUID, nodeID, node | Total number of active/in-progress cloud group snapshots from this node (Active, Queued states). |
| px_node_stats_volume_replica_count | cluster, clusterUUID, nodeID, node | Total number of volume replicas hosted on this node. |
Node status metrics
| Name | Labels | Description |
|---|---|---|
| px_node_status_node_status | cluster, clusterUUID, nodeID, node | Status of this node (https://libopenstorage.github.io/w/master.generated-api/#status) |
| px_node_status_license_expiry | cluster, clusterUUID, nodeID, node | Number of days until License (or License lease) expires (<0 means Expired) |
Storage pool metrics
| Name | Labels | Description |
|---|---|---|
| px_pool_stats_pool_written_bytes | cluster, clusterUUID, nodeID, node, pool | Bytes written since last interval for this pool. Deprecated. |
| px_pool_stats_pool_write_latency_seconds | cluster, clusterUUID, nodeID, node, pool | Average time spent per write operation for this pool. Deprecated. |
| px_pool_stats_pool_writethroughput | cluster, clusterUUID, nodeID, node, pool | Average number of bytes written per second for this pool. Deprecated. |
| px_pool_stats_pool_flushed_bytes | cluster, clusterUUID, nodeID, node, pool | Number of flushed bytes since last interval for this pool. Deprecated. |
| px_pool_stats_pool_num_flushes | cluster, clusterUUID, nodeID, node, pool | Number of flush(sync) operations since last interval for this pool. Deprecated. |
| px_pool_stats_pool_flushms | cluster, clusterUUID, nodeID, node, pool | Latency for flush for this pool. Deprecated. |
| px_pool_stats_pool_provisioned_bytes | cluster, clusterUUID, nodeID, node, pool | Provisioned storage space in bytes for this pool. Deprecated. |
| px_pool_stats_pool_status | cluster, clusterUUID, nodeID, node, pool | Status of this Pool (0=Offline,1=Online). Deprecated. |
| px_pool_stats_num_reads | cluster, clusterUUID, nodeID, node, pool | Number of read operations in the last interval for this pool |
| px_pool_stats_read_bytes | cluster, clusterUUID, nodeID, node, pool | Read bytes in the last interval for this pool |
| px_pool_stats_read_ms | cluster, clusterUUID, nodeID, node, pool | Total time in milliseconds spent reading in the last interval for this pool |
| px_pool_stats_read_iops | cluster, clusterUUID, nodeID, node, pool | Average number of completed read operations per second for this pool |
| px_pool_stats_readthroughput | cluster, clusterUUID, nodeID, node, pool | Average number of bytes read per second for this pool |
| px_pool_stats_read_latency_seconds | cluster, clusterUUID, nodeID, node, pool | Average time spent per read operation for this pool |
| px_pool_stats_written_bytes | cluster, clusterUUID, nodeID, node, pool | Bytes written since last interval for this pool |
| px_pool_stats_num_writes | cluster, clusterUUID, nodeID, node, pool | Number of write operations in the last interval for this pool |
| px_pool_stats_write_ms | cluster, clusterUUID, nodeID, node, pool | Total time in milliseconds spent writing in the last interval for this pool |
| px_pool_stats_write_latency_seconds | cluster, clusterUUID, nodeID, node, pool | Average time spent per write operation for this pool |
| px_pool_stats_write_iops | cluster, clusterUUID, nodeID, node, pool | Average number of completed write operations per second for this pool |
| px_pool_stats_writethroughput | cluster, clusterUUID, nodeID, node, pool | Average number of bytes written per second for this pool |
| px_pool_stats_num_discards | cluster, clusterUUID, nodeID, node, pool | Number of discard operations in the last interval for this pool |
| px_pool_stats_discarded_bytes | cluster, clusterUUID, nodeID, node, pool | Bytes discarded since last interval for this pool |
| px_pool_stats_discard_ms | cluster, clusterUUID, nodeID, node, pool | Total time in milliseconds spent discarding in the last interval for this pool |
| px_pool_stats_discard_iops | cluster, clusterUUID, nodeID, node, pool | Average number of completed discard operations per second for this pool |
| px_pool_stats_discardthroughput | cluster, clusterUUID, nodeID, node, pool | Average number of bytes discarded per second for this pool |
| px_pool_stats_discard_latency_seconds | cluster, clusterUUID, nodeID, node, pool | Average time spent per discard operation for this pool |
| px_pool_stats_flushed_bytes | cluster, clusterUUID, nodeID, node, pool | Number of flushed bytes since last interval for this pool |
| px_pool_stats_num_flushes | cluster, clusterUUID, nodeID, node, pool | Number of flush(sync) operations since last interval for this pool |
| px_pool_stats_flushms | cluster, clusterUUID, nodeID, node, pool | Latency for flush for this pool |
| px_pool_stats_flush_iops | cluster, clusterUUID, nodeID, node, pool | Average number of completed flush operations per second for this pool |
| px_pool_stats_flushthroughput | cluster, clusterUUID, nodeID, node, pool | Average number of bytes flushed per second for this pool |
| px_pool_stats_flush_latency_seconds | cluster, clusterUUID, nodeID, node, pool | Average time spent per flush operation for this pool |
| px_pool_stats_num_ios | cluster, clusterUUID, nodeID, node, pool | Number of IO operations in the last interval for this pool |
| px_pool_stats_io_bytes | cluster, clusterUUID, nodeID, node, pool | IO bytes in the last interval for this pool |
| px_pool_stats_io_ms | cluster, clusterUUID, nodeID, node, pool | Total time in milliseconds spent doing IO in the last interval for this pool |
| px_pool_stats_io_iops | cluster, clusterUUID, nodeID, node, pool | Average number of completed IO operations per second for this pool |
| px_pool_stats_iothroughput | cluster, clusterUUID, nodeID, node, pool | Average number of bytes IO per second for this pool |
| px_pool_stats_io_latency_seconds | cluster, clusterUUID, nodeID, node, pool | Average time spent per IO operation for this pool |
| px_pool_stats_provisioned_bytes | cluster, clusterUUID, nodeID, node, pool | Provisioned storage space in bytes for this pool |
| px_pool_stats_status | cluster, clusterUUID, nodeID, node, pool | Status of this Pool (0=Offline,1=Online,2=Full,3=NotFound,4=Maintenance,5=Degraded,6=Background activity) |
| px_pool_stats_available_bytes | cluster, clusterUUID, nodeID, node, pool | Available storage space in bytes for this pool |
| px_pool_stats_used_bytes | cluster, clusterUUID, nodeID, node, pool | Used storage space in bytes for this pool |
| px_pool_stats_total_bytes | cluster, clusterUUID, nodeID, node, pool | Total storage space in bytes for this pool |
| px_pool_stats_written_bytes_total | cluster, clusterUUID, nodeID, node, pool | Total bytes written for this pool |
| px_pool_stats_flushed_bytes_total | cluster, clusterUUID, nodeID, node, pool | Total number of flushed bytes |
| px_pool_stats_num_flushes_total | cluster, clusterUUID, nodeID, node, pool | Total number of flush(sync) operations |
| px_pool_stats_flushms_total | cluster, clusterUUID, nodeID, node, pool | Total time spent in flush |
Process metrics
| Name | Labels | Description |
|---|---|---|
| px_proc_stats_virt | cluster, clusterUUID, nodeID, node, proc | Virtual memory in bytes |
| px_proc_stats_res | cluster, clusterUUID, nodeID, node, proc | Resident set size memory in bytes |
| px_proc_stats_cputime | cluster, clusterUUID, nodeID, node, proc | Amount of time that this process has been scheduled in user and kernel mode measured in clock ticks |
Portworx cache metrics
| Name | Labels | Description |
|---|---|---|
| px_px_cache_status | cluster, clusterUUID, nodeID, node, pool | Cache enabled (0=No,1=Yes) |
| px_px_cache_total_blocks | cluster, clusterUUID, nodeID, node, pool | Number of total blocks in the cache |
| px_px_cache_used_blocks | cluster, clusterUUID, nodeID, node, pool | Number of used blocks in the cache |
| px_px_cache_dirty_blocks | cluster, clusterUUID, nodeID, node, pool | Number of dirty blocks in the cache |
| px_px_cache_read_hits | cluster, clusterUUID, nodeID, node, pool | Number of read hits for the cache |
| px_px_cache_read_miss | cluster, clusterUUID, nodeID, node, pool | Number of read misses for the cache |
| px_px_cache_write_hits | cluster, clusterUUID, nodeID, node, pool | Number of write hits for the cache |
| px_px_cache_write_miss | cluster, clusterUUID, nodeID, node, pool | Number of write misses for the cache |
| px_px_cache_block_size | cluster, clusterUUID, nodeID, node, pool | Block size for the cache |
| px_px_cache_mode | cluster, clusterUUID, nodeID, node, pool | Mode of the cache |
| px_px_cache_migrate_promote | cluster, clusterUUID, nodeID, node, pool | Number of blocks promoted to the cache |
| px_px_cache_migrate_demote | cluster, clusterUUID, nodeID, node, pool | Number of block demoted from the cache |
| px_px_cache_io_mbps | cluster, clusterUUID, nodeID, node, pool | Approx cache bandwidth from cache internal actions |
Rebalance metrics
| Name | Labels | Description |
|---|---|---|
| px_rebalance_rebalance_job_state | cluster, clusterUUID, nodeID, node | Rebalance job state (0 = pending, 1 = running, 2 = done, 3 = paused, 4 = cancelled) |
| px_rebalance_provision_space_rebalanced_bytes_total | cluster, clusterUUID, nodeID, node | Total provisioned space rebalanced (only counts add (since remove has equal value as add)) |
| px_rebalance_used_space_rebalanced_bytes_total | cluster, clusterUUID, nodeID, node | Total used space rebalanced (only counts add (since remove has equal value as add)) |
| px_rebalance_volumes_rebalanced_total | cluster, clusterUUID, nodeID, node | Total volumes affected by rebalance operation |
| px_rebalance_overloaded_pools_total | cluster, clusterUUID, nodeID, node | Number of overloaded pools |
Volume metrics
| Name | Labels | Description |
|---|---|---|
| px_volume_usage_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Used storage space in bytes for this volume |
| px_volume_capacity_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Configured size in bytes for this volume |
| px_volume_halevel | cluster, clusterUUID, nodeID, node, volume, volumename | Configured HA level for this volume |
| px_volume_currhalevel | cluster, clusterUUID, nodeID, node, volume, volumename | Current HA level for this volume |
| px_volume_iopriority | cluster, clusterUUID, nodeID, node, volume, volumename | Configured IO priority for this volume |
| px_volume_elapsed_time_since_detached_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Seconds elapsed since the volume is detached |
| px_volume_elapsed_time_since_attached_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Seconds elapsed since the volume is attached |
| px_volume_attached | cluster, clusterUUID, nodeID, node, volume, volumename | Attached state for this volume (0=detached,1=attached) |
| px_volume_status | cluster, clusterUUID, nodeID, node, volume, volumename | Status for this volume (https://libopenstorage.github.io/w/master.generated-api/#volumestatus) |
| px_volume_state | cluster, clusterUUID, nodeID, node, volume, volumename | State for this volume (https://libopenstorage.github.io/w/master.generated-api/#volumestate) |
| px_volume_attached_state | cluster, clusterUUID, nodeID, node, volume, volumename | Attached state for this volume (valid only if volume is attached) (https://libopenstorage.github.io/w/master.generated-api/#attachstate) |
| px_volume_fs_health_status | cluster, clusterUUID, nodeID, node, volume, volumename | Filesystem health status for this volume (https://libopenstorage.github.io/w/master.generated-api/#filesystemhealthstatus) |
| px_volume_replication_status | cluster, clusterUUID, nodeID, node, volume, volumename | Replication Status for this volume (0 : up, 1 : not in quorum, 2 : resync state, 3 : degraded, 4 : detached, 5 : restore) |
| px_volume_fs_usage_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Used storage space in bytes as reported by the filesystem for this volume |
| px_volume_fs_capacity_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Total storage space in bytes as reported by the filesystem for this volume |
| px_volume_vol_read_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully read bytes during this interval for this volume. Deprecated. |
| px_volume_vol_written_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully written bytes during this interval for this volume. Deprecated. |
| px_volume_vol_reads | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully completed read operations during this interval for this volume. Deprecated. |
| px_volume_vol_writes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully completed write operations during this interval for this volume. Deprecated. |
| px_volume_read_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully read bytes during this interval for this volume |
| px_volume_written_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully written bytes during this interval for this volume |
| px_volume_reads | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully completed read operations during this interval for this volume |
| px_volume_writes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully completed write operations during this interval for this volume |
| px_volume_read_bytes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully read bytes for this volume |
| px_volume_written_bytes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully written bytes for this volume |
| px_volume_reads_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully completed read operations for this volume |
| px_volume_writes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully completed write operations for this volume |
| px_volume_iops | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successful completed I/O operations per second during this interval for this volume |
| px_volume_write_iops | cluster, clusterUUID, nodeID, node, volume, volumename | Average number of completed write operations per second for this volume |
| px_volume_read_iops | cluster, clusterUUID, nodeID, node, volume, volumename | Average number of completed read operations per second for this volume |
| px_volume_vol_num_sequential_writes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of sequential write I/O operations during this interval for this volume |
| px_volume_vol_num_sequential_reads | cluster, clusterUUID, nodeID, node, volume, volumename | Number of sequential read I/O operations during this interval for this volume |
| px_volume_vol_num_random_writes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of random write I/O operations during this interval for this volume |
| px_volume_vol_num_random_reads | cluster, clusterUUID, nodeID, node, volume, volumename | Number of random read I/O operations during this interval for this volume |
| px_volume_depth_io | cluster, clusterUUID, nodeID, node, volume, volumename | Number of I/O operations currently in progress for this volume |
| px_volume_readthroughput | cluster, clusterUUID, nodeID, node, volume, volumename | Number of bytes read per second during this interval for this volume |
| px_volume_writethroughput | cluster, clusterUUID, nodeID, node, volume, volumename | Number of bytes written per second during this interval for this volume |
| px_volume_vol_bytes_reclaimed | cluster, clusterUUID, nodeID, node, volume, volumename | Number of bytes reclaimed by fstrim operation |
| px_volume_vol_bytes_reclaimable | cluster, clusterUUID, nodeID, node, volume, volumename | Number of bytes reclaimable on the volume as seen by fstrim operation |
| px_volume_vol_read_latency_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Average time spent per successfully completed read operation in seconds during this interval for this volume. Deprecated. |
| px_volume_vol_write_latency_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Average time spent per successfully completed write operation in seconds during this interval for this volume. Deprecated. |
| px_volume_read_latency_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Average time spent per successfully completed read operation in seconds for this volume |
| px_volume_write_latency_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Average time spent per successfully completed write operation in seconds for this volume |
| px_volume_num_long_reads | cluster, clusterUUID, nodeID, node, volume, volumename | Number of long reads for this volume |
| px_volume_num_long_writes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of long writes for this volume |
| px_volume_num_long_flushes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of long flushes for this volume |
| px_volume_num_db_flushes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of DB flushes for this volume |
| px_volume_replica_read_bytes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully read bytes for this replica volume |
| px_volume_replica_written_bytes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully written bytes for this replica volume |
| px_volume_replica_reads_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully completed read operations for this replica volume |
| px_volume_replica_writes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully completed write operations for this replica volume |
| px_volume_replica_flushes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully completed flush operations for this replica volume |
| px_volume_replica_read_ms_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total time spent doing read operations in milliseconds for this replica volume |
| px_volume_replica_write_ms_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total time doing write operations in milliseconds for this replica volume |
| px_volume_replica_flush_ms_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total time doing write operations in milliseconds for this replica volume |
| px_volume_dev_depth_io | cluster, clusterUUID, nodeID, node, volume, volumename | Number of I/O operations currently in progress as reported by the kernel pxd device for this volume |
| px_volume_dev_writethroughput | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully written bytes per second as reported by the kernel pxd device for this volume |
| px_volume_dev_readthroughput | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully read bytes per second as reported by the kernel pxd device for this volume |
| px_volume_dev_read_latency_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Average time spent per successfully completed read in seconds as reported by the kernel pxd device for this volume |
| px_volume_dev_write_latency_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Average time spent per successfully completed write in seconds as reported by the kernel pxd device for this volume |
| px_volume_dev_read_bytes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully read bytes as reported by the kernel pxd device for this volume |
| px_volume_dev_written_bytes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully written bytes as reported by the kernel pxd device for this volume |
| px_volume_dev_reads_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully completed read operations as reported by the kernel pxd device for this volume |
| px_volume_dev_writes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully completed write operations as reported by the kernel pxd device for this volume |
| px_volume_dev_read_seconds_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total time spent reading in seconds for this disk as reported by the kernel pxd device for this volume |
| px_volume_dev_write_seconds_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total time spent writing in seconds for this disk as reported by the kernel pxd device for this volume |
| px_volume_unique_blocks | cluster, clusterUUID, nodeID, node, volume, volumename | Size(in bytes) of unique blocks for this volume |
| px_volume_timestamp_records | cluster, clusterUUID, nodeID, node, volume, volumename | Number of timestamp records accumulated |
| px_volume_timestamp_records_per_node | cluster, clusterUUID, nodeID, node, volume, volumename | Number of timestamp records accumulated for a node |
| px_volume_num_discard_ops | cluster, clusterUUID, nodeID, node, volume, volumename | Number of discard operations for this volume |
| px_volume_discarded_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of discarded bytes for this volume |
Autopilot metrics
The following Autopilot Prometheus metrics are available starting with Portworx Operator version 25.6.0.
| Name | Description |
|---|---|
| autopilot_errors_total | Total number of autopilot errors |
| autopilot_action_failed_count | Total number of declined or failed actions |
| autopilot_action_failed_total | Total number of failed actions per rule object |
| autopilot_objects_in_stable_condition_total | Total number of objects with stable conditions per rule |
| autopilot_action_taken_total | Total number of actions taken per rule object |
| autopilot_objects_total | Total number of objects per rule |
| autopilot_objects_with_rules_total | Total number of objects matching any rules |
| autopilot_rules_in_system | Total number of rules in the system |
The following Autopilot Datadog metrics are available starting with Autopilot version 1.5.0.
| Name | Description |
|---|---|
| autopilot_datadog_empty_response_total | Total number of Datadog queries that returned empty results |
| autopilot_datadog_query_validation_failures_total | Total number of Datadog query validation failures |
| autopilot_datadog_failures_total | Total number of Datadog query failures by type |
| autopilot_datadog_success_total | Total number of successful Datadog queries |
The following Autopilot metrics are available starting with Autopilot version 1.6.0.
| Name | Description |
|---|---|
| autopilot_object_evaluation_duration_seconds | Time taken to evaluate managed objects and process their state transitions. |
| autopilot_engine_evaluation_duration_seconds | Time taken to discover and process Autopilot rules in the cluster. |
| autopilot_prom_auth_token_refresh_total | Total number of successful Prometheus authentication token refreshes. |
| autopilot_prom_auth_token_refresh_failures_total | Total number of failed Prometheus authentication token refreshes. |
| autopilot_action_blocked | Boolean that indicates if an action is blocked by another action (1: Blocked, 0: Not Blocked) |
Portworx DR and Stork metrics
The following Prometheus metrics are available starting with Stork version 25.6.0.
Migration metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
| stork_migration_status | Gauge | cluster, name, namespace, clusterpair, schedule, status | Migration status—used to filter and list migrations by status within the cluster. |
| stork_migration_stage | Gauge | name, namespace, schedule | Stage of migration (0 = Initial, 1 = PreExecRule, 2 = PostExecRule, 3 = Volumes, 4 = Applications, 5 = Final). |
| stork_migration_duration | Gauge | cluster, name, namespace, schedule, stage | Duration of migration in seconds. Stage label values: volume (volume migration) or resource (resource migration). |
| stork_migration_duration_seconds | Histogram | cluster, namespace, clusterpair, schedule, stage | Total time taken by a migration in seconds. |
| stork_migration_resources | Gauge | cluster, name, namespace, clusterpair, schedule, state | Count of resources collected for migration (state: planned or migrated). |
| stork_migration_volumes | Gauge | cluster, name, namespace, clusterpair, schedule, state | Volumes planned for migration (state: planned or migrated). |
| stork_migration_created_total | Counter | cluster, namespace, schedule, clusterpair | Increments when a migration is created. |
| stork_migration_completed_total | Counter | cluster, namespace, schedule, status, clusterpair | Increments when a migration reaches a terminal state (Successful, Failed, or PartialSuccess). |
| stork_migration_transfer_rate | Gauge | cluster, name, namespace, clusterpair, schedule | Data transfer rate for a single migration (bytes/second). |
| stork_migration_transferred_bytes | Histogram | cluster, namespace, schedule, clusterpair | Total bytes transferred during a single migration. |
| stork_migration_completion_timestamp_seconds | Gauge | cluster, namespace, schedule, status, clusterpair | Timestamp when the latest migration completed for a schedule with the status being Successful, PartialSuccess, or Failed. |
Migration schedule metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
| stork_migration_schedule_status | Gauge | cluster, name, namespace, clusterpair, status | Status of migration schedules. |
| stork_migration_schedule_last_run_status | Gauge | cluster, name, namespace, clusterpair, status | Status of the latest run of each migration schedule. |
Action metrics (Failover/Failback)
| Metric | Type | Labels | Description |
|---|---|---|---|
| stork_action_status | Gauge | cluster, type, name, namespace, schedule, status | Status of Failover or Failback actions. |
| stork_action_created_total | Counter | cluster, type, namespace, schedule | Increments when a Failover/Failback action is created. |
| stork_action_completed_total | Counter | cluster, type, namespace, schedule, status | Increments when a Failover/Failback action reaches a terminal state. |
| stork_action_completion_timestamp_seconds | Gauge | cluster, type, name, namespace, schedule, status | Timestamp when a Failover/Failback action completes with the status as Successful, Failed, RollbackSuccessful, or RollbackFailed. |
ClusterPair metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
| stork_clusterpair_scheduler_status | Gauge | name, namespace | Status of the cluster pair's health with respect to Kubernetes (0 = NotCreated, 1 = NotReady, 2 = Ready, 3 = Error, 4 = Degraded, 5 = Deleting, 6 = NotProvided, 7 = Paused, 8 = Unknown, 9 = MaintenanceMode). |
| stork_clusterpair_storage_status | Gauge | name, namespace | Status of the cluster pair's health with respect to storage provisioner (0 = NotCreated, 1 = NotReady, 2 = Ready, 3 = Error, 4 = Degraded, 5 = Deleting, 6 = NotProvided, 7 = Paused, 8 = Unknown, 9 = MaintenanceMode). |
Namespace metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
| stork_all_namespaces | Gauge | cluster | Number of namespaces in the Kubernetes cluster, including Kubernetes system namespaces. |
| stork_protected_namespaces | Gauge | cluster | Count of namespaces being migrated through active migration schedules. |
Stork Scheduler metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
| stork_hyperconverged_pods_total | Gauge | pod, namespace | Number of pods scheduled on a node where replicas for all pod volumes exist (hyper-converged by the Stork scheduler). |
| stork_non_hyperconverged_pods_total | Gauge | pod, namespace | Number of pods not hyper-converged by the Stork scheduler. |
| stork_semi_hyperconverged_pods_total | Gauge | pod, namespace | Number of pods partially hyper-converged (scheduled on nodes with some but not all volume replicas). |
| stork_pods_rescheduled_total | Counter | The total number of pods rescheduled by the Stork Pod Monitor. |
Notes
- Stork Scheduler metrics are exported from all Stork pods because scheduling requests can reach any pod.
- DR metrics are exported only from the Stork leader pod.
- For metrics where status or stage is a label (such as
stork_migration_statusorstork_action_status), the label represents the actual state, and the metric value is always1when present.