Skip to main content

Internal KVDB for Portworx on Kubernetes

The built-in internal KVDB is enabled by default when you install Portworx through Portworx Central. Portworx automatically deploys the internal KVDB cluster and manages the internal key-value store cluster. The internal KVDB is installed on a set of three nodes in your cluster, and it removes the requirement for an external KVDB.


Internal KVDB requires a dedicated drive to store its data. You can choose to have either a KVDB drive or a metadata drive.

KVDB drive:

  • If IOPS are independent of disk size, the minimum recommended size is 32 GB and a minimum of 450 IOPs.
  • If IOPS are dependent on disk size, the recommended size is 150 GB to ensure you get a minimum of 450 IOPs.

Metadata drive:

  • If IOPS are independent of disk size, then the recommended size is 64 GB and a minimum of 450 IOPs.
  • If IOPS are dependent on disk size, then the recommended size is 150 GB to ensure you get a minimum of 450 IOPS.

If you are using a cloud drive, you should size it in accordance with your cloud provider's specifications to achieve the minimum required IOPs.

(Optional) Label the nodes before installing Portworx

If you want to control where KVDB is placed, you can use the following command to label nodes in Kubernetes. This command will designate them for the internal KVDB cluster:

kubectl label nodes <list-of-node-names> px/metadata-node=true

Here are a few scenarios based on the labels and their values:

  • If a node is labeled px/metadata-node=true, it becomes part of the internal KVDB cluster.
  • If a node is labeled px/metadata-node=false, it will not be part of the internal KVDB cluster.
  • If no node labels are found, all storage nodes have the potential to run the internal KVDB
  • If an incorrect label is present on the node, such as px/metadata-node=blah, Portworx will not start KVDB on that node.
  • If no node is labeled as px/metadata-node=true, but one node is labeled as px/metadata-node=false, that specific node will never be part of the KVDB cluster, whereas the remaining nodes are potential internal KVDB nodes.

Install Portworx using the internal KVDB cluster

Navigate to Portworx Central and select Built-in for etcd when creating your Portworx cluster.

Additional concepts

Here are a few concepts related to the internal KVDB.

How internal KVDB cluster recovery works

If a Portworx node that was previously part of the internal KVDB cluster becomes unavailable for more than 3 minutes, any other available storage nodes that are not part of the KVDB cluster will attempt to join it. The following is the sequence of actions taken when a node attempts to rejoin the KVDB cluster:

  1. The non-responsive member is removed from the internal KVDB cluster. (Note that the node remains part of the Portworx cluster.)
  2. Internal KVDB is initiated on a new node.
  3. The new node adds itself to the internal KVDB cluster.

Internal KVDB backup mechanism

Portworx takes regular internal backups of its key-value space and saves them in a file on the host. Portworx takes these internal KVDB backups every 2 minutes and places them in the /var/lib/osd/kvdb_backup directory on all nodes in your cluster. It also keeps a rolling count of 10 backups at a time on the internal KVDB storage drive.

A backup file looks as follows:

ls /var/lib/osd/kvdb_backup

These backup files can be used for recovering the internal KVDB cluster in the event of a disaster.


In the event of a cluster failure, such as:

  • When all internal KVDB nodes and the corresponding drives where their data resides are lost and unrecoverable.
  • When quorum is lost (two internal KVDB nodes are lost and are unrecoverable). You should contact Portworx Technical Support to recover your cluster from such a state.
Was this page helpful?