Portworx Metrics for monitoring
Alert metrics
Alert metrics use variable labels depending on the resource type. Each metric has a label named after its resource type (such as node, volume, drive, or cluster) that contains the resource ID, plus a clusterid label.
| Name | Labels | Description |
|---|---|---|
| px_alerts_callhomefailure | node, clusterid | Increments when a call-home failure alert is raised; decrements when cleared |
| px_alerts_nodestatechange | node, clusterid | Increments when a node state-change alert is raised; decrements when cleared |
| px_alerts_nodestartsuccess | node, clusterid | Increments when a node successfully starts; decrements when cleared |
Backup metrics
Per-backup metrics include volume and volumename labels that identify the backed-up volume. Node-level transfer metrics (px_backup_stats_node_*) do not include volume labels.
| Name | Labels | Description |
|---|---|---|
| px_backup_stats_backup_status | cluster, clusterUUID, nodeID, node, volume, volumename | Status for this backup (0=InProgress,1=Done,2=Failed) |
| px_backup_stats_backup_size | cluster, clusterUUID, nodeID, node, volume, volumename | Size in bytes for this backup |
| px_backup_stats_backup_duration_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Duration in seconds for this backup |
| px_backup_stats_backup_uploaded_bytes_mbs | cluster, clusterUUID, nodeID, node, volume, volumename | Delta bytes uploaded in MB/s from last interval for this backup |
| px_backup_stats_download_size_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Size of downloaded bytes during backup/restore from cloud |
| px_backup_stats_upload_size_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Size of uploaded bytes during backup/restore from cloud |
| px_backup_stats_get_apis_invoked | cluster, clusterUUID, nodeID, node, volume, volumename | Number of times GET API was invoked |
| px_backup_stats_put_apis_invoked | cluster, clusterUUID, nodeID, node, volume, volumename | Number of times PUT API was invoked |
| px_backup_stats_read_time_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Time spent in read operations during cloudsnap backup |
| px_backup_stats_write_time_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Time spent in write operations during cloudsnap restore |
| px_backup_stats_upload_time_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Time spent in upload operations during cloudsnap backup |
| px_backup_stats_download_time_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Time spent in download operations during cloudsnap restore |
| px_backup_stats_compress_time_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Time spent in compression during cloudsnap backup |
| px_backup_stats_encrypt_time_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Time spent in encryption during cloudsnap backup |
| px_backup_stats_node_upload_bytes_total | cluster, clusterUUID, nodeID, node | Total bytes uploaded from this node across all cloudsnaps |
| px_backup_stats_node_download_bytes_total | cluster, clusterUUID, nodeID, node | Total bytes downloaded to this node across all cloudsnaps |
| px_backup_stats_node_upload_bytes_interval | cluster, clusterUUID, nodeID, node | Bytes uploaded from this node in the current interval |
| px_backup_stats_node_download_bytes_interval | cluster, clusterUUID, nodeID, node | Bytes downloaded to this node in the current interval |
| px_backup_stats_node_upload_rate_bytes_per_sec | cluster, clusterUUID, nodeID, node | Current upload rate in bytes per second for this node |
| px_backup_stats_node_download_rate_bytes_per_sec | cluster, clusterUUID, nodeID, node | Current download rate in bytes per second for this node |
Cluster metrics
| Name | Labels | Description |
|---|---|---|
| px_cluster_cpu_percent | cluster, clusterUUID, nodeID, node | Percentage of CPU Used |
| px_cluster_memory_utilized_percent | cluster, clusterUUID, nodeID, node | Percentage of memory utilization |
| px_cluster_disk_total_bytes | cluster, clusterUUID, nodeID, node | Total storage space in bytes for this node |
| px_cluster_disk_available_bytes | cluster, clusterUUID, nodeID, node | Available storage space in bytes for this node |
| px_cluster_disk_utilized_bytes | cluster, clusterUUID, nodeID, node | Utilized storage space in bytes for this node |
| px_cluster_pendingio | cluster, clusterUUID, nodeID, node | Number of read and write operations currently in progress for this node |
| px_cluster_trashcan_volumes | cluster, clusterUUID, nodeID, node | Number of volumes currently in the trashcan |
Cluster status metrics
| Name | Labels | Description |
|---|---|---|
| px_cluster_status_cluster_size | cluster, clusterUUID, nodeID, node | Node count for your portworx cluster. Deprecated. |
| px_cluster_status_size | cluster, clusterUUID, nodeID, node | Node count for your portworx cluster |
| px_cluster_status_cluster_quorum | cluster, clusterUUID, nodeID, node | Indicates if the cluster is in quorum. Deprecated. |
| px_cluster_status_quorum | cluster, clusterUUID, nodeID, node | Indicates if the cluster is in quorum |
| px_cluster_status_nodes_online | cluster, clusterUUID, nodeID, node | Number of online nodes in the cluster (includes storage and storageless) |
| px_cluster_status_nodes_offline | cluster, clusterUUID, nodeID, node | Number of offline nodes in the cluster (includes storage and storageless) |
| px_cluster_status_nodes_storage_down | cluster, clusterUUID, nodeID, node | Number of nodes where the storage is full or down |
| px_cluster_status_storage_nodes_online | cluster, clusterUUID, nodeID, node | Number of storage nodes that are online |
| px_cluster_status_storage_nodes_offline | cluster, clusterUUID, nodeID, node | Number of storage nodes that are offline |
| px_cluster_status_storage_nodes_decommissioned | cluster, clusterUUID, nodeID, node | Number of storage nodes that are decommissioned |
Device delete metrics
| Name | Labels | Description |
|---|---|---|
| px_device_delete_delete_after_discard_enabled | cluster, clusterUUID, nodeID, node, poolid | Current configuration value indicating if NBDD is enabled |
| px_device_delete_discard_bytes_total | cluster, clusterUUID, nodeID, node, poolid | Total bytes discarded across all device deletes |
| px_device_delete_discard_duration_msec_total | cluster, clusterUUID, nodeID, node, poolid | Total time spent in discard operations in milliseconds across all device deletes |
| px_device_delete_discard_errors_total | cluster, clusterUUID, nodeID, node, poolid | Total number of discard errors encountered across all device deletes |
| px_device_delete_discard_operations_total | cluster, clusterUUID, nodeID, node, poolid | Total number of discard operations performed across all device deletes |
| px_device_delete_discard_size_bytes | cluster, clusterUUID, nodeID, node, poolid | Current configuration value for discard size in bytes |
| px_device_delete_operations_completed_total | cluster, clusterUUID, nodeID, node, poolid | Total number of delete operations completed |
| px_device_delete_operations_inprogress | cluster, clusterUUID, nodeID, node, poolid | Current number of delete operations in progress |
| px_device_delete_operations_queued | cluster, clusterUUID, nodeID, node, poolid | Current number of delete operations queued |
| px_device_delete_release_bytes_total | cluster, clusterUUID, nodeID, node, poolid | Total size of devices processed for storage release in bytes |
| px_device_delete_release_duration_msec_total | cluster, clusterUUID, nodeID, node, poolid | Total time spent in storage release operations in milliseconds |
| px_device_delete_release_operations_total | cluster, clusterUUID, nodeID, node, poolid | Total number of devices processed for storage release |
Disk metrics
| Name | Labels | Description |
|---|---|---|
| px_disk_stats_used_bytes | cluster, clusterUUID, nodeID, node, disk, pool | Total storage in bytes for this disk |
| px_disk_stats_interval_seconds | cluster, clusterUUID, nodeID, node, disk, pool | interval_seconds |
| px_disk_stats_io_seconds | cluster, clusterUUID, nodeID, node, disk, pool | Time spent doing IO in seconds for this disk |
| px_disk_stats_progress_io | cluster, clusterUUID, nodeID, node, disk, pool | IO's currently in progress for this disk |
| px_disk_stats_disk_read_bytes | cluster, clusterUUID, nodeID, node, disk, pool | Total bytes read for this disk. Deprecated. |
| px_disk_stats_read_bytes | cluster, clusterUUID, nodeID, node, disk, pool | Total bytes read for this disk |
| px_disk_stats_write_bytes_seconds | cluster, clusterUUID, nodeID, node, disk, pool | Total written bytes for this disk. Deprecated. |
| px_disk_stats_written_bytes | cluster, clusterUUID, nodeID, node, disk, pool | Total written bytes for this disk |
| px_disk_stats_read_seconds | cluster, clusterUUID, nodeID, node, disk, pool | Total time spent reading in seconds for this disk |
| px_disk_stats_write_seconds | cluster, clusterUUID, nodeID, node, disk, pool | Total time spent writing in seconds for this disk |
| px_disk_stats_read_latency_seconds | cluster, clusterUUID, nodeID, node, disk, pool | Average time spent per read operation in seconds for this disk |
| px_disk_stats_write_latency | cluster, clusterUUID, nodeID, node, disk, pool | Average time spent per write operation in seconds for this disk. Deprecated. |
| px_disk_stats_write_latency_seconds | cluster, clusterUUID, nodeID, node, disk, pool | Average time spent per write operation in seconds |
| px_disk_stats_disk_num_reads | cluster, clusterUUID, nodeID, node, disk, pool | Total number of read operations completed successfully for this disk. Deprecated. |
| px_disk_stats_disk_num_writes | cluster, clusterUUID, nodeID, node, disk, pool | Total number of write operations completed successfully for this disk. Deprecated. |
| px_disk_stats_num_reads | cluster, clusterUUID, nodeID, node, disk, pool | Total number of read operations completed successfully for this disk |
| px_disk_stats_num_writes | cluster, clusterUUID, nodeID, node, disk, pool | Total number of write operations completed successfully for this disk |
| px_disk_stats_num_reads_total | cluster, clusterUUID, nodeID, node, disk, pool | Total number of read operations completed successfully for this disk |
| px_disk_stats_num_writes_total | cluster, clusterUUID, nodeID, node, disk, pool | Total number of write operations completed successfully for this disk |
| px_disk_stats_written_bytes_total | cluster, clusterUUID, nodeID, node, disk, pool | Total bytes written for this disk |
| px_disk_stats_read_bytes_total | cluster, clusterUUID, nodeID, node, disk, pool | Total bytes read for this disk |
| px_disk_stats_read_seconds_total | cluster, clusterUUID, nodeID, node, disk, pool | Total time spent reading in seconds for this disk |
| px_disk_stats_write_seconds_total | cluster, clusterUUID, nodeID, node, disk, pool | Total time spent writing in seconds for this disk |
FA/FB API metrics
| Name | Labels | Description |
|---|---|---|
| px_fafb_apis_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to all the configured FA/FB endpoints |
| px_fafb_apis_array_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /arrays endpoint. |
| px_fafb_apis_login_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /login endpoint. |
| px_fafb_apis_controllers_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /controllers endpoint |
| px_fafb_apis_alerts_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /alerts endpoint |
| px_fafb_apis_ports_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /ports endpoint |
| px_fafb_apis_volumes_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /volumes endpoint |
| px_fafb_apis_volumesnapshots_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /volume-snapshots endpoint |
| px_fafb_apis_hosts_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /hosts endpoint |
| px_fafb_apis_connections_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /connections endpoint |
| px_fafb_apis_version_requests_total | cluster, clusterUUID, nodeID, node | Total number of API requests made to the /api_version endpoint |
KVDB metrics
| Name | Labels | Description |
|---|---|---|
| px_kvdb_get_requests_total | cluster, clusterUUID, nodeID, node | Total number of get requests for a key |
| px_kvdb_snapshot_requests_total | cluster, clusterUUID, nodeID, node | Total number of kvdb snapshot requests for a list of prefixes |
| px_kvdb_put_requests_total | cluster, clusterUUID, nodeID, node | Total number of put requests for a key |
| px_kvdb_create_requests_total | cluster, clusterUUID, nodeID, node | Total number of create requests for a key |
| px_kvdb_update_requests_total | cluster, clusterUUID, nodeID, node | Total number of update requests for a key |
| px_kvdb_enumerate_requests_total | cluster, clusterUUID, nodeID, node | Total number of enumerate requests for a key |
| px_kvdb_delete_requests_total | cluster, clusterUUID, nodeID, node | Total number of delete requests for a key |
| px_kvdb_keys_requests_total | cluster, clusterUUID, nodeID, node | Total number of list key requests for a prefix |
| px_kvdb_cas_requests_total | cluster, clusterUUID, nodeID, node | Total number of compare and sets for a key |
| px_kvdb_cad_requests_total | cluster, clusterUUID, nodeID, node | Total number of compare and deletes for a key |
| px_kvdb_lock_requests_total | cluster, clusterUUID, nodeID, node | Total number of lock requests for a key |
| px_kvdb_unlock_requests_total | cluster, clusterUUID, nodeID, node | Total number of unlock requests for a key |
| px_kvdb_watchkey_requests_total | cluster, clusterUUID, nodeID, node | Total number of watch requests for a key from a node |
| px_kvdb_watchtree_requests_total | cluster, clusterUUID, nodeID, node | Total number of watch requests for a prefix from a node |
| px_kvdb_adduser_requests_total | cluster, clusterUUID, nodeID, node | Total number of add user requests |
| px_kvdb_removeuser_requests_total | cluster, clusterUUID, nodeID, node | Total number of remove user requests |
| px_kvdb_grantuser_access_requests_total | cluster, clusterUUID, nodeID, node | Total number of grant access requests for a user |
| px_kvdb_revokeuser_access_requests_total | cluster, clusterUUID, nodeID, node | Total number of revoke access requests for a user |
| px_kvdb_addmember_requests_total | cluster, clusterUUID, nodeID, node | Total number of add member requests from a node |
| px_kvdb_addlearner_requests_total | cluster, clusterUUID, nodeID, node | Total number of add learner requests from a node |
| px_kvdb_removemember_requests_total | cluster, clusterUUID, nodeID, node | Total number of remove member requests from a node |
| px_kvdb_updatemember_requests_total | cluster, clusterUUID, nodeID, node | Total number of update member requests from a node |
| px_kvdb_listmembers_requests_total | cluster, clusterUUID, nodeID, node | Total number of list member requests from a node |
| px_kvdb_setendpoints_requests_total | cluster, clusterUUID, nodeID, node | Total number of set endpoint requests from a node |
| px_kvdb_getendpoints_requests_total | cluster, clusterUUID, nodeID, node | Total number of get endpoint requests from a node |
| px_kvdb_defragment_requests_total | cluster, clusterUUID, nodeID, node | Total number of defragment requests from a node |
| px_kvdb_watch_latency_seconds | cluster, clusterUUID, nodeID, node | Time taken in seconds between a kvdb put and a corresponding watch update |
KVDB health metrics
| Name | Labels | Description |
|---|---|---|
| px_kvdb_health_state_node_view | cluster, clusterUUID, nodeID, node | This node's kvdb state (1 = healthy, 2 = not healthy) |
| px_kvdb_health_state_cluster_view | cluster, clusterUUID, nodeID, node | This node's view of other node's kvdb state (1 = healthy, 2 = not healthy) |
License metrics
| Name | Labels | Description |
|---|---|---|
| px_license_expiry | cluster, clusterUUID, nodeID, node | License expiry in days. A negative value indicates the license has expired. |
| px_license_next_refresh | cluster, clusterUUID, nodeID, node | Next license refresh timestamp in minutes. A negative value indicates that the license will not be refreshed. |
| px_license_type | cluster, clusterUUID, nodeID, node | Indicates the type of license (0=PX-Developer, 1=Legacy Enterprise, 2=Trial, 3=PX-Enterprise, 4=PX-Enterprise floating, 5=IBM Cloud Pak for Data, 6=CISCO OEM, 7=PX-Essential, 8=Portworx CSI for FA/FB, 9=INVALID LICENSE) |
Network I/O metrics
| Name | Labels | Description |
|---|---|---|
| px_network_io_bytessent | cluster, clusterUUID, nodeID, node | Number of bytes sent during this interval |
| px_network_io_received_bytes | cluster, clusterUUID, nodeID, node | Number of bytes received during this interval |
| px_network_io_sent_bytes_total | cluster, clusterUUID, nodeID, node | Total number of bytes sent |
| px_network_io_received_bytes_total | cluster, clusterUUID, nodeID, node | Total number of bytes received |
| px_network_io_reservation_waits | cluster, clusterUUID, nodeID, node, nodeid | Total number of SM waits for reservation |
| px_network_io_reservation_wake_ups | cluster, clusterUUID, nodeID, node, nodeid | Total number of SM wake ups after getting the reservation |
NFS pool metrics
| Name | Labels | Description |
|---|---|---|
| px_nfs_pool_stats_packets_arrived | cluster, clusterUUID, nodeID, node | Counts the number of NFS packets that arrived on the socket. |
| px_nfs_pool_stats_sockets_enqueued | cluster, clusterUUID, nodeID, node | Counts how many times an NFS transport is enqueued to wait for an nfsd thread to service it, i.e. no nfsd thread was considered available. |
| px_nfs_pool_stats_threads_woken | cluster, clusterUUID, nodeID, node | Counts how many times an idle nfsd thread is woken to try to receive some data from an NFS transport. |
| px_nfs_pool_stats_threads_timedout | cluster, clusterUUID, nodeID, node | Counts how many times an nfsd thread triggered an idle timeout, i.e. was not woken to handle any incoming network packets for some time. |
Node metrics
| Name | Labels | Description |
|---|---|---|
| px_node_stats_free_mem | cluster, clusterUUID, nodeID, node | Available memory in bytes |
| px_node_stats_total_mem | cluster, clusterUUID, nodeID, node | Total memory in bytes |
| px_node_stats_used_mem | cluster, clusterUUID, nodeID, node | Used memory in bytes |
| px_node_stats_cpu_usage | cluster, clusterUUID, nodeID, node | Percent of CPU consumption |
| px_node_stats_busy_threads_percent | cluster, clusterUUID, nodeID, node | Percentage of Busy Threads |
| px_node_stats_thread_pool_high_priority_queue_size | cluster, clusterUUID, nodeID, node | Thread Pool High Priority Queue Size |
| px_node_stats_thread_pool_long_running_busy_threads | cluster, clusterUUID, nodeID, node | Long Running Thread Pool Number Of Busy Threads |
| px_node_stats_thread_pool_long_running_queue_size | cluster, clusterUUID, nodeID, node | Long Running Thread Pool Pending Items In Queue |
| px_node_stats_thread_pool_long_running_max_threads | cluster, clusterUUID, nodeID, node | Long Running Thread Pool Count Of Max Threads |
| px_node_stats_writeback_list_num_dirty_pools | cluster, clusterUUID, nodeID, node | Writeback List Number of Dirty Pools |
| px_node_stats_writeback_list_num_pending_log_bufs | cluster, clusterUUID, nodeID, node | Writeback List Number of Pending Log Buffers |
| px_node_stats_writeback_list_low_cos_queue_size | cluster, clusterUUID, nodeID, node | Writeback List Low Cos Queue Size |
| px_node_stats_writeback_list_high_cos_queue_size | cluster, clusterUUID, nodeID, node | Writeback List High Cos Queue Size |
| px_node_stats_sm_run_queue_size | cluster, clusterUUID, nodeID, node | Statemachine Run Queue Size |
| px_node_stats_kaio_pending_queue_size | cluster, clusterUUID, nodeID, node | Kaio Pending Queue Size |
| px_node_stats_request_scheduler_queue_size | cluster, clusterUUID, nodeID, node | Request Scheduler Queue Size |
| px_node_stats_num_reads | cluster, clusterUUID, nodeID, node | Number of read operations in the last interval for this node |
| px_node_stats_read_bytes | cluster, clusterUUID, nodeID, node | Read bytes in the last interval for this node |
| px_node_stats_read_ms | cluster, clusterUUID, nodeID, node | Total time in milliseconds spent reading in the last interval for this node |
| px_node_stats_read_iops | cluster, clusterUUID, nodeID, node | Average number of completed read operations per second for this node |
| px_node_stats_readthroughput | cluster, clusterUUID, nodeID, node | Average number of bytes read per second for this node |
| px_node_stats_read_latency_seconds | cluster, clusterUUID, nodeID, node | Average time spent per read operation for this node |
| px_node_stats_num_writes | cluster, clusterUUID, nodeID, node | Number of write operations in the last interval for this node |
| px_node_stats_written_bytes | cluster, clusterUUID, nodeID, node | Bytes written since last interval for this node |
| px_node_stats_write_ms | cluster, clusterUUID, nodeID, node | Total time in milliseconds spent writing in the last interval for this node |
| px_node_stats_write_iops | cluster, clusterUUID, nodeID, node | Average number of completed write operations per second for this node |
| px_node_stats_writethroughput | cluster, clusterUUID, nodeID, node | Average number of bytes written per second for this node |
| px_node_stats_write_latency_seconds | cluster, clusterUUID, nodeID, node | Average time spent per write operation for this node |
| px_node_stats_num_discards | cluster, clusterUUID, nodeID, node | Number of discard operations in the last interval for this node |
| px_node_stats_discarded_bytes | cluster, clusterUUID, nodeID, node | Bytes discarded since last interval for this node |
| px_node_stats_discard_ms | cluster, clusterUUID, nodeID, node | Total time in milliseconds spent discarding in the last interval for this node |
| px_node_stats_discard_iops | cluster, clusterUUID, nodeID, node | Average number of completed discard operations per second for this node |
| px_node_stats_discardthroughput | cluster, clusterUUID, nodeID, node | Average number of bytes discarded per second for this node |
| px_node_stats_discard_latency_seconds | cluster, clusterUUID, nodeID, node | Average time spent per discard operation for this node |
| px_node_stats_num_flushes | cluster, clusterUUID, nodeID, node | Number of flush(sync) operations in the last interval for this node |
| px_node_stats_flushed_bytes | cluster, clusterUUID, nodeID, node | Number of flushed bytes since last interval for this node |
| px_node_stats_flushms | cluster, clusterUUID, nodeID, node | Latency for flush for this node |
| px_node_stats_flush_iops | cluster, clusterUUID, nodeID, node | Average number of completed flush operations per second for this node |
| px_node_stats_flushthroughput | cluster, clusterUUID, nodeID, node | Average number of bytes flushed per second for this node |
| px_node_stats_flush_latency_seconds | cluster, clusterUUID, nodeID, node | Average time spent per flush operation for this node |
| px_node_stats_num_ios | cluster, clusterUUID, nodeID, node | Number of IO operations in the last interval for this node |
| px_node_stats_io_bytes | cluster, clusterUUID, nodeID, node | IO bytes in the last interval for this node |
| px_node_stats_io_ms | cluster, clusterUUID, nodeID, node | Total time in milliseconds spent doing IO in the last interval for this node |
| px_node_stats_io_iops | cluster, clusterUUID, nodeID, node | Average number of completed IO operations per second for this node |
| px_node_stats_iothroughput | cluster, clusterUUID, nodeID, node | Average number of bytes IO per second for this node |
| px_node_stats_io_latency_seconds | cluster, clusterUUID, nodeID, node | Average time spent per IO operation for this node |
| px_node_stats_procfs_mem_available_bytes | cluster, clusterUUID, nodeID, node | Available memory in bytes |
| px_node_stats_procfs_mem_dirty_bytes | cluster, clusterUUID, nodeID, node | The total amount of memory waiting to be written back to the disk |
| px_node_stats_procfs_mem_writeback_bytes | cluster, clusterUUID, nodeID, node | The total amount of memory actively being written back to the disk |
| px_node_stats_procfs_mem_total_bytes | cluster, clusterUUID, nodeID, node | Total amount of usable RAM which is physical RAM minus a number of reserved bits and the kernel binary code |
| px_node_stats_procfs_mem_free_bytes | cluster, clusterUUID, nodeID, node | The amount of physical RAM, in bytes, left unused by the system. |
| px_node_stats_procfs_mem_buffers_bytes | cluster, clusterUUID, nodeID, node | The amount, in bytes, of temporary storage for raw disk blocks |
| px_node_stats_procfs_mem_cached_bytes | cluster, clusterUUID, nodeID, node | The amount of physical RAM, in bytes, used as cache memory |
| px_node_stats_procfs_mem_active_anon_bytes | cluster, clusterUUID, nodeID, node | The amount of anonymous and tmpfs/shmem memory, in bytes, that is in active use, or was in active use since the last time the system moved something to swap |
| px_node_stats_procfs_mem_inactive_anon_bytes | cluster, clusterUUID, nodeID, node | The amount of anonymous and tmpfs/shmem memory, in bytes, that is a candidate for eviction |
| px_node_stats_procfs_mem_active_file_bytes | cluster, clusterUUID, nodeID, node | The amount of file cache memory, in bytes, that is in active use, or was in active use since the last time the system reclaimed memory |
| px_node_stats_procfs_mem_inactive_file_bytes | cluster, clusterUUID, nodeID, node | The amount of file cache memory, in bytes, that is newly loaded from the disk, or is a candidate for reclaiming |
| px_node_stats_procfs_mem_unevictable_bytes | cluster, clusterUUID, nodeID, node | The amount of memory, in bytes, discovered by the pageout code, that is not evictable because it is locked into memory by user programs |
| px_node_stats_procfs_mem_mlocked_bytes | cluster, clusterUUID, nodeID, node | The total amount of memory, in bytes, that is not evictable because it is locked into memory by user programs |
| px_node_stats_procfs_mem_anon_pages_bytes | cluster, clusterUUID, nodeID, node | The total amount of memory, in bytes, used by pages that are not backed by files and are mapped into userspace page tables |
| px_node_stats_procfs_mem_mapped_bytes | cluster, clusterUUID, nodeID, node | The memory, in bytes, used for files that have been mmaped, such as libraries |
| px_node_stats_procfs_mem_sh_mem_bytes | cluster, clusterUUID, nodeID, node | The total amount of memory, in bytes, used by shared memory (shmem) and tmpfs |
| px_node_stats_num_skinnysnaps | cluster, clusterUUID, nodeID, node | The number of SkinnySnaps |
| px_node_stats_skinnysnaps_num_repls_skips | cluster, clusterUUID, nodeID, node | The number of target replicas skipped for a snapshot due to SkinnySnap |
| px_node_stats_skinnysnaps_num_repls_snapshots | cluster, clusterUUID, nodeID, node | The number of target replicas that underwent a snapshot operation due to SkinnySnap |
| px_node_stats_relaxed_reclaim_pending | cluster, clusterUUID, nodeID, node | The number of volumes in RelaxedReclaim pending queue |
| px_node_stats_relaxed_reclaim_skipped | cluster, clusterUUID, nodeID, node | The number of volume deletes that were skipped from being delayed even when RelaxedReclaim was enabled. |
| px_node_stats_relaxed_reclaim_deleted | cluster, clusterUUID, nodeID, node | The number of volume deletes done through RelaxedReclaim |
| px_node_stats_num_volumes | cluster, clusterUUID, nodeID, node | The number of volumes hosted on this node |
Node status metrics
| Name | Labels | Description |
|---|---|---|
| px_node_status_node_status | cluster, clusterUUID, nodeID, node | Status of this node (https://libopenstorage.github.io/w/master.generated-api/#status) |
| px_node_status_license_expiry | cluster, clusterUUID, nodeID, node | Number of days until License (or License lease) expires (<0 means Expired) |
Storage pool metrics
| Name | Labels | Description |
|---|---|---|
| px_pool_stats_pool_written_bytes | cluster, clusterUUID, nodeID, node, pool | Bytes written since last interval for this pool. Deprecated. |
| px_pool_stats_pool_write_latency_seconds | cluster, clusterUUID, nodeID, node, pool | Average time spent per write operation for this pool. Deprecated. |
| px_pool_stats_pool_writethroughput | cluster, clusterUUID, nodeID, node, pool | Average number of bytes written per second for this pool. Deprecated. |
| px_pool_stats_pool_flushed_bytes | cluster, clusterUUID, nodeID, node, pool | Number of flushed bytes since last interval for this pool. Deprecated. |
| px_pool_stats_pool_num_flushes | cluster, clusterUUID, nodeID, node, pool | Number of flush(sync) operations since last interval for this pool. Deprecated. |
| px_pool_stats_pool_flushms | cluster, clusterUUID, nodeID, node, pool | Latency for flush for this pool. Deprecated. |
| px_pool_stats_pool_provisioned_bytes | cluster, clusterUUID, nodeID, node, pool | Provisioned storage space in bytes for this pool. Deprecated. |
| px_pool_stats_pool_status | cluster, clusterUUID, nodeID, node, pool | Status of this Pool (0=Offline,1=Online). Deprecated. |
| px_pool_stats_num_reads | cluster, clusterUUID, nodeID, node, pool | Number of read operations in the last interval for this pool |
| px_pool_stats_read_bytes | cluster, clusterUUID, nodeID, node, pool | Read bytes in the last interval for this pool |
| px_pool_stats_read_ms | cluster, clusterUUID, nodeID, node, pool | Total time in milliseconds spent reading in the last interval for this pool |
| px_pool_stats_read_iops | cluster, clusterUUID, nodeID, node, pool | Average number of completed read operations per second for this pool |
| px_pool_stats_readthroughput | cluster, clusterUUID, nodeID, node, pool | Average number of bytes read per second for this pool |
| px_pool_stats_read_latency_seconds | cluster, clusterUUID, nodeID, node, pool | Average time spent per read operation for this pool |
| px_pool_stats_written_bytes | cluster, clusterUUID, nodeID, node, pool | Bytes written since last interval for this pool |
| px_pool_stats_num_writes | cluster, clusterUUID, nodeID, node, pool | Number of write operations in the last interval for this pool |
| px_pool_stats_write_ms | cluster, clusterUUID, nodeID, node, pool | Total time in milliseconds spent writing in the last interval for this pool |
| px_pool_stats_write_latency_seconds | cluster, clusterUUID, nodeID, node, pool | Average time spent per write operation for this pool |
| px_pool_stats_write_iops | cluster, clusterUUID, nodeID, node, pool | Average number of completed write operations per second for this pool |
| px_pool_stats_writethroughput | cluster, clusterUUID, nodeID, node, pool | Average number of bytes written per second for this pool |
| px_pool_stats_num_discards | cluster, clusterUUID, nodeID, node, pool | Number of discard operations in the last interval for this pool |
| px_pool_stats_discarded_bytes | cluster, clusterUUID, nodeID, node, pool | Bytes discarded since last interval for this pool |
| px_pool_stats_discard_ms | cluster, clusterUUID, nodeID, node, pool | Total time in milliseconds spent discarding in the last interval for this pool |
| px_pool_stats_discard_iops | cluster, clusterUUID, nodeID, node, pool | Average number of completed discard operations per second for this pool |
| px_pool_stats_discardthroughput | cluster, clusterUUID, nodeID, node, pool | Average number of bytes discarded per second for this pool |
| px_pool_stats_discard_latency_seconds | cluster, clusterUUID, nodeID, node, pool | Average time spent per discard operation for this pool |
| px_pool_stats_flushed_bytes | cluster, clusterUUID, nodeID, node, pool | Number of flushed bytes since last interval for this pool |
| px_pool_stats_num_flushes | cluster, clusterUUID, nodeID, node, pool | Number of flush(sync) operations since last interval for this pool |
| px_pool_stats_flushms | cluster, clusterUUID, nodeID, node, pool | Latency for flush for this pool |
| px_pool_stats_flush_iops | cluster, clusterUUID, nodeID, node, pool | Average number of completed flush operations per second for this pool |
| px_pool_stats_flushthroughput | cluster, clusterUUID, nodeID, node, pool | Average number of bytes flushed per second for this pool |
| px_pool_stats_flush_latency_seconds | cluster, clusterUUID, nodeID, node, pool | Average time spent per flush operation for this pool |
| px_pool_stats_num_ios | cluster, clusterUUID, nodeID, node, pool | Number of IO operations in the last interval for this pool |
| px_pool_stats_io_bytes | cluster, clusterUUID, nodeID, node, pool | IO bytes in the last interval for this pool |
| px_pool_stats_io_ms | cluster, clusterUUID, nodeID, node, pool | Total time in milliseconds spent doing IO in the last interval for this pool |
| px_pool_stats_io_iops | cluster, clusterUUID, nodeID, node, pool | Average number of completed IO operations per second for this pool |
| px_pool_stats_iothroughput | cluster, clusterUUID, nodeID, node, pool | Average number of bytes IO per second for this pool |
| px_pool_stats_io_latency_seconds | cluster, clusterUUID, nodeID, node, pool | Average time spent per IO operation for this pool |
| px_pool_stats_provisioned_bytes | cluster, clusterUUID, nodeID, node, pool | Provisioned storage space in bytes for this pool |
| px_pool_stats_status | cluster, clusterUUID, nodeID, node, pool | Status of this Pool (0=Offline,1=Online,2=Full,3=NotFound,4=Maintenance,5=Degraded,6=Background activity) |
| px_pool_stats_available_bytes | cluster, clusterUUID, nodeID, node, pool | Available storage space in bytes for this pool |
| px_pool_stats_used_bytes | cluster, clusterUUID, nodeID, node, pool | Used storage space in bytes for this pool |
| px_pool_stats_total_bytes | cluster, clusterUUID, nodeID, node, pool | Total storage space in bytes for this pool |
| px_pool_stats_written_bytes_total | cluster, clusterUUID, nodeID, node, pool | Total bytes written for this pool |
| px_pool_stats_flushed_bytes_total | cluster, clusterUUID, nodeID, node, pool | Total number of flushed bytes |
| px_pool_stats_num_flushes_total | cluster, clusterUUID, nodeID, node, pool | Total number of flush(sync) operations |
| px_pool_stats_flushms_total | cluster, clusterUUID, nodeID, node, pool | Total time spent in flush |
Process metrics
| Name | Labels | Description |
|---|---|---|
| px_proc_stats_virt | cluster, clusterUUID, nodeID, node, proc | Virtual memory in bytes |
| px_proc_stats_res | cluster, clusterUUID, nodeID, node, proc | Resident set size memory in bytes |
| px_proc_stats_cputime | cluster, clusterUUID, nodeID, node, proc | Amount of time that this process has been scheduled in user and kernel mode measured in clock ticks |
Portworx cache metrics
| Name | Labels | Description |
|---|---|---|
| px_px_cache_status | cluster, clusterUUID, nodeID, node, pool | Cache enabled (0=No,1=Yes) |
| px_px_cache_total_blocks | cluster, clusterUUID, nodeID, node, pool | Number of total blocks in the cache |
| px_px_cache_used_blocks | cluster, clusterUUID, nodeID, node, pool | Number of used blocks in the cache |
| px_px_cache_dirty_blocks | cluster, clusterUUID, nodeID, node, pool | Number of dirty blocks in the cache |
| px_px_cache_read_hits | cluster, clusterUUID, nodeID, node, pool | Number of read hits for the cache |
| px_px_cache_read_miss | cluster, clusterUUID, nodeID, node, pool | Number of read misses for the cache |
| px_px_cache_write_hits | cluster, clusterUUID, nodeID, node, pool | Number of write hits for the cache |
| px_px_cache_write_miss | cluster, clusterUUID, nodeID, node, pool | Number of write misses for the cache |
| px_px_cache_block_size | cluster, clusterUUID, nodeID, node, pool | Block size for the cache |
| px_px_cache_mode | cluster, clusterUUID, nodeID, node, pool | Mode of the cache |
| px_px_cache_migrate_promote | cluster, clusterUUID, nodeID, node, pool | Number of blocks promoted to the cache |
| px_px_cache_migrate_demote | cluster, clusterUUID, nodeID, node, pool | Number of block demoted from the cache |
| px_px_cache_io_mbps | cluster, clusterUUID, nodeID, node, pool | Approx cache bandwidth from cache internal actions |
Rebalance metrics
| Name | Labels | Description |
|---|---|---|
| px_rebalance_rebalance_job_state | cluster, clusterUUID, nodeID, node | Rebalance job state (0 = pending, 1 = running, 2 = done, 3 = paused, 4 = cancelled) |
| px_rebalance_provision_space_rebalanced_bytes_total | cluster, clusterUUID, nodeID, node | Total provisioned space rebalanced (only counts add (since remove has equal value as add)) |
| px_rebalance_used_space_rebalanced_bytes_total | cluster, clusterUUID, nodeID, node | Total used space rebalanced (only counts add (since remove has equal value as add)) |
| px_rebalance_volumes_rebalanced_total | cluster, clusterUUID, nodeID, node | Total volumes affected by rebalance operation |
| px_rebalance_overloaded_pools_total | cluster, clusterUUID, nodeID, node | Number of overloaded pools |
Volume metrics
| Name | Labels | Description |
|---|---|---|
| px_volume_usage_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Used storage space in bytes for this volume |
| px_volume_capacity_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Configured size in bytes for this volume |
| px_volume_halevel | cluster, clusterUUID, nodeID, node, volume, volumename | Configured HA level for this volume |
| px_volume_currhalevel | cluster, clusterUUID, nodeID, node, volume, volumename | Current HA level for this volume |
| px_volume_iopriority | cluster, clusterUUID, nodeID, node, volume, volumename | Configured IO priority for this volume |
| px_volume_elapsed_time_since_detached_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Seconds elapsed since the volume is detached |
| px_volume_elapsed_time_since_attached_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Seconds elapsed since the volume is attached |
| px_volume_attached | cluster, clusterUUID, nodeID, node, volume, volumename | Attached state for this volume (0=detached,1=attached) |
| px_volume_status | cluster, clusterUUID, nodeID, node, volume, volumename | Status for this volume (https://libopenstorage.github.io/w/master.generated-api/#volumestatus) |
| px_volume_state | cluster, clusterUUID, nodeID, node, volume, volumename | State for this volume (https://libopenstorage.github.io/w/master.generated-api/#volumestate) |
| px_volume_attached_state | cluster, clusterUUID, nodeID, node, volume, volumename | Attached state for this volume (valid only if volume is attached) (https://libopenstorage.github.io/w/master.generated-api/#attachstate) |
| px_volume_fs_health_status | cluster, clusterUUID, nodeID, node, volume, volumename | Filesystem health status for this volume (https://libopenstorage.github.io/w/master.generated-api/#filesystemhealthstatus) |
| px_volume_replication_status | cluster, clusterUUID, nodeID, node, volume, volumename | Replication Status for this volume (0 : up, 1 : not in quorum, 2 : resync state, 3 : degraded, 4 : detached, 5 : restore) |
| px_volume_fs_usage_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Used storage space in bytes as reported by the filesystem for this volume |
| px_volume_fs_capacity_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Total storage space in bytes as reported by the filesystem for this volume |
| px_volume_vol_read_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully read bytes during this interval for this volume. Deprecated. |
| px_volume_vol_written_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully written bytes during this interval for this volume. Deprecated. |
| px_volume_vol_reads | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully completed read operations during this interval for this volume. Deprecated. |
| px_volume_vol_writes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully completed write operations during this interval for this volume. Deprecated. |
| px_volume_read_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully read bytes during this interval for this volume |
| px_volume_written_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully written bytes during this interval for this volume |
| px_volume_reads | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully completed read operations during this interval for this volume |
| px_volume_writes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully completed write operations during this interval for this volume |
| px_volume_read_bytes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully read bytes for this volume |
| px_volume_written_bytes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully written bytes for this volume |
| px_volume_reads_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully completed read operations for this volume |
| px_volume_writes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully completed write operations for this volume |
| px_volume_iops | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successful completed I/O operations per second during this interval for this volume |
| px_volume_write_iops | cluster, clusterUUID, nodeID, node, volume, volumename | Average number of completed write operations per second for this volume |
| px_volume_read_iops | cluster, clusterUUID, nodeID, node, volume, volumename | Average number of completed read operations per second for this volume |
| px_volume_vol_num_sequential_writes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of sequential write I/O operations during this interval for this volume |
| px_volume_vol_num_sequential_reads | cluster, clusterUUID, nodeID, node, volume, volumename | Number of sequential read I/O operations during this interval for this volume |
| px_volume_vol_num_random_writes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of random write I/O operations during this interval for this volume |
| px_volume_vol_num_random_reads | cluster, clusterUUID, nodeID, node, volume, volumename | Number of random read I/O operations during this interval for this volume |
| px_volume_depth_io | cluster, clusterUUID, nodeID, node, volume, volumename | Number of I/O operations currently in progress for this volume |
| px_volume_readthroughput | cluster, clusterUUID, nodeID, node, volume, volumename | Number of bytes read per second during this interval for this volume |
| px_volume_writethroughput | cluster, clusterUUID, nodeID, node, volume, volumename | Number of bytes written per second during this interval for this volume |
| px_volume_vol_bytes_reclaimed | cluster, clusterUUID, nodeID, node, volume, volumename | Number of bytes reclaimed by fstrim operation |
| px_volume_vol_bytes_reclaimable | cluster, clusterUUID, nodeID, node, volume, volumename | Number of bytes reclaimable on the volume as seen by fstrim operation |
| px_volume_vol_read_latency_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Average time spent per successfully completed read operation in seconds during this interval for this volume. Deprecated. |
| px_volume_vol_write_latency_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Average time spent per successfully completed write operation in seconds during this interval for this volume. Deprecated. |
| px_volume_read_latency_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Average time spent per successfully completed read operation in seconds for this volume |
| px_volume_write_latency_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Average time spent per successfully completed write operation in seconds for this volume |
| px_volume_num_long_reads | cluster, clusterUUID, nodeID, node, volume, volumename | Number of long reads for this volume |
| px_volume_num_long_writes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of long writes for this volume |
| px_volume_num_long_flushes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of long flushes for this volume |
| px_volume_num_db_flushes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of DB flushes for this volume |
| px_volume_replica_read_bytes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully read bytes for this replica volume |
| px_volume_replica_written_bytes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully written bytes for this replica volume |
| px_volume_replica_reads_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully completed read operations for this replica volume |
| px_volume_replica_writes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully completed write operations for this replica volume |
| px_volume_replica_flushes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully completed flush operations for this replica volume |
| px_volume_replica_read_ms_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total time spent doing read operations in milliseconds for this replica volume |
| px_volume_replica_write_ms_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total time doing write operations in milliseconds for this replica volume |
| px_volume_replica_flush_ms_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total time doing write operations in milliseconds for this replica volume |
| px_volume_dev_depth_io | cluster, clusterUUID, nodeID, node, volume, volumename | Number of I/O operations currently in progress as reported by the kernel pxd device for this volume |
| px_volume_dev_writethroughput | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully written bytes per second as reported by the kernel pxd device for this volume |
| px_volume_dev_readthroughput | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully read bytes per second as reported by the kernel pxd device for this volume |
| px_volume_dev_read_latency_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Average time spent per successfully completed read in seconds as reported by the kernel pxd device for this volume |
| px_volume_dev_write_latency_seconds | cluster, clusterUUID, nodeID, node, volume, volumename | Average time spent per successfully completed write in seconds as reported by the kernel pxd device for this volume |
| px_volume_dev_read_bytes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully read bytes as reported by the kernel pxd device for this volume |
| px_volume_dev_written_bytes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Number of successfully written bytes as reported by the kernel pxd device for this volume |
| px_volume_dev_reads_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully completed read operations as reported by the kernel pxd device for this volume |
| px_volume_dev_writes_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total number of successfully completed write operations as reported by the kernel pxd device for this volume |
| px_volume_dev_read_seconds_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total time spent reading in seconds for this disk as reported by the kernel pxd device for this volume |
| px_volume_dev_write_seconds_total | cluster, clusterUUID, nodeID, node, volume, volumename | Total time spent writing in seconds for this disk as reported by the kernel pxd device for this volume |
| px_volume_unique_blocks | cluster, clusterUUID, nodeID, node, volume, volumename | Size(in bytes) of unique blocks for this volume |
| px_volume_timestamp_records | cluster, clusterUUID, nodeID, node, volume, volumename | Number of timestamp records accumulated |
| px_volume_timestamp_records_per_node | cluster, clusterUUID, nodeID, node, volume, volumename | Number of timestamp records accumulated for a node |
| px_volume_num_discard_ops | cluster, clusterUUID, nodeID, node, volume, volumename | Number of discard operations for this volume |
| px_volume_discarded_bytes | cluster, clusterUUID, nodeID, node, volume, volumename | Number of discarded bytes for this volume |
Autopilot metrics
The following Autopilot Prometheus metrics are available starting with Portworx Operator version 25.6.0.
| Name | Description |
|---|---|
| autopilot_errors_total | Total number of autopilot errors |
| autopilot_action_failed_count | Total number of declined or failed actions |
| autopilot_action_failed_total | Total number of failed actions per rule object |
| autopilot_objects_in_stable_condition_total | Total number of objects with stable conditions per rule |
| autopilot_action_taken_total | Total number of actions taken per rule object |
| autopilot_objects_total | Total number of objects per rule |
| autopilot_objects_with_rules_total | Total number of objects matching any rules |
| autopilot_rules_in_system | Total number of rules in the system |
The following Autopilot Datadog metrics are available starting with Autopilot version 1.5.0.
| Name | Description |
|---|---|
| autopilot_datadog_empty_response_total | Total number of Datadog queries that returned empty results |
| autopilot_datadog_query_validation_failures_total | Total number of Datadog query validation failures |
| autopilot_datadog_failures_total | Total number of Datadog query failures by type |
| autopilot_datadog_success_total | Total number of successful Datadog queries |
Portworx DR and Stork metrics
The following Prometheus metrics are available starting with Stork version 25.6.0.
Migration metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
| stork_migration_status | Gauge | cluster, name, namespace, clusterpair, schedule, status | Migration status—used to filter and list migrations by status within the cluster. |
| stork_migration_stage | Gauge | name, namespace, schedule | Stage of migration (0 = Initial, 1 = PreExecRule, 2 = PostExecRule, 3 = Volumes, 4 = Applications, 5 = Final). |
| stork_migration_duration | Gauge | cluster, name, namespace, schedule, stage | Duration of migration in seconds. Stage label values: volume (volume migration) or resource (resource migration). |
| stork_migration_duration_seconds | Histogram | cluster, namespace, clusterpair, schedule, stage | Total time taken by a migration in seconds. |
| stork_migration_resources | Gauge | cluster, name, namespace, clusterpair, schedule, state | Count of resources collected for migration (state: planned or migrated). |
| stork_migration_volumes | Gauge | cluster, name, namespace, clusterpair, schedule, state | Volumes planned for migration (state: planned or migrated). |
| stork_migration_created_total | Counter | cluster, namespace, schedule, clusterpair | Increments when a migration is created. |
| stork_migration_completed_total | Counter | cluster, namespace, schedule, status, clusterpair | Increments when a migration reaches a terminal state (Successful, Failed, or PartialSuccess). |
| stork_migration_transfer_rate | Gauge | cluster, name, namespace, clusterpair, schedule | Data transfer rate for a single migration (bytes/second). |
| stork_migration_transferred_bytes | Histogram | cluster, namespace, schedule, clusterpair | Total bytes transferred during a single migration. |
| stork_migration_completion_timestamp_seconds | Gauge | cluster, namespace, schedule, status, clusterpair | Timestamp when the latest migration completed for a schedule with the status being Successful, PartialSuccess, or Failed. |
Migration schedule metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
| stork_migration_schedule_status | Gauge | cluster, name, namespace, clusterpair, status | Status of migration schedules. |
| stork_migration_schedule_last_run_status | Gauge | cluster, name, namespace, clusterpair, status | Status of the latest run of each migration schedule. |
Action metrics (Failover/Failback)
| Metric | Type | Labels | Description |
|---|---|---|---|
| stork_action_status | Gauge | cluster, type, name, namespace, schedule, status | Status of Failover or Failback actions. |
| stork_action_created_total | Counter | cluster, type, namespace, schedule | Increments when a Failover/Failback action is created. |
| stork_action_completed_total | Counter | cluster, type, namespace, schedule, status | Increments when a Failover/Failback action reaches a terminal state. |
| stork_action_completion_timestamp_seconds | Gauge | cluster, type, name, namespace, schedule, status | Timestamp when a Failover/Failback action completes with the status as Successful, Failed, RollbackSuccessful, or RollbackFailed. |
ClusterPair metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
| stork_clusterpair_scheduler_status | Gauge | name, namespace | Status of the cluster pair's health with respect to Kubernetes (0 = NotCreated, 1 = NotReady, 2 = Ready, 3 = Error, 4 = Degraded, 5 = Deleting, 6 = NotProvided, 7 = Paused, 8 = Unknown, 9 = MaintenanceMode). |
| stork_clusterpair_storage_status | Gauge | name, namespace | Status of the cluster pair's health with respect to storage provisioner (0 = NotCreated, 1 = NotReady, 2 = Ready, 3 = Error, 4 = Degraded, 5 = Deleting, 6 = NotProvided, 7 = Paused, 8 = Unknown, 9 = MaintenanceMode). |
Namespace metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
| stork_all_namespaces | Gauge | cluster | Number of namespaces in the Kubernetes cluster, including Kubernetes system namespaces. |
| stork_protected_namespaces | Gauge | cluster | Count of namespaces being migrated through active migration schedules. |
Stork Scheduler metrics
| Metric | Type | Labels | Description |
|---|---|---|---|
| stork_hyperconverged_pods_total | Gauge | pod, namespace | Number of pods scheduled on a node where replicas for all pod volumes exist (hyper-converged by the Stork scheduler). |
| stork_non_hyperconverged_pods_total | Gauge | pod, namespace | Number of pods not hyper-converged by the Stork scheduler. |
| stork_semi_hyperconverged_pods_total | Gauge | pod, namespace | Number of pods partially hyper-converged (scheduled on nodes with some but not all volume replicas). |
| stork_pods_rescheduled_total | Counter | The total number of pods rescheduled by the Stork Pod Monitor. |
Notes
- Stork Scheduler metrics are exported from all Stork pods because scheduling requests can reach any pod.
- DR metrics are exported only from Stork the leader pod.
- For metrics where status or stage is a label (such as
stork_migration_statusorstork_action_status), the label represents the actual state, and the metric value is always1when present.