Skip to main content
Version: 3.5

External KVDB for Portworx on Kubernetes

Portworx can use an external key-value database (KVDB) to store cluster metadata and configuration data. Portworx supports etcd as the only external KVDB. etcd is a highly available, distributed key-value store that you deploy and manage independently of Portworx. Use an external KVDB only when required by your deployment.

Prerequisites​

Before configuring Portworx to use an external KVDB, ensure that your etcd cluster meets the following requirements:

  • A three node etcd cluster with etcd version 3.3 or later.

  • A minimum of 8 GB of memory dedicated to each etcd node.

  • KVDB drive requirements:

    • If IOPS are independent of disk size, Portworx recommends a minimum size of 32 GB or a minimum of 450 IOPs.
    • If IOPS are dependent on disk size, Portworx recommends a size of 150 GB.
    note

    If you use cloud-based storage, size the drive according to your cloud provider’s specifications to meet the minimum IOPS requirements.

  • The recommended hardware requirements.

Best practices for maintaining an etcd cluster​

Follow these best practices to keep your etcd cluster healthy and performant when used with Portworx.

Configure Compaction​

etcd stores a complete history of its keyspace, and regular compaction is necessary to avoid performance issues and running out of storage space. Regular compaction helps manage the memory usage of the etcd process.

You can configure the compaction period using the etcd tool with the --auto-compaction-retention option.

Portworx recommeds that you retain the history for the last 3 hours. To achieve this, set the --auto-compaction-retention value to 3. For information on how to set the compaction period, see the etcd documentation.

Configure Space Quota​

A space quota, or setting a database size limit in etcd, is crucial for maintaining a stable cluster. Without a space quota, etcd can experience performance issues when the keyspace grows excessively and the available storage space is consumed, resulting in unpredictable cluster behavior.

Portworx recommeds that you set the space quota to a maximum value of 8 GiB. To achieve this, use the etcd tool to set the --quota-backend-bytes to $((8*1024*1024*1024))

For more information on how to set the space quota, see the etcd documentation.

Take Regular Snapshots​

You can recover your etcd cluster in the event of a disaster by taking a snapshot of its keyspace. Run the following command as part of a cron job, either on the etcd nodes or on a separate node where you want to store these periodic etcd snapshots:

ETCDCTL_API=3 etcdctl --endpoints="<comma-separated-etcd-url>" snapshot save </path/to/snapshot-file> --command-timeout=60s

The above example command will store etcd snapshots to the specified file and if any etcd operation takes longer than 60 seconds, it will time out and be canceled.

For more information on how to setup a recovery mechanism, see the etcd documentation.

Secure your etcd communication​

You can configure and secure etcd communication in your environment by storing SSL certificates as Kubernetes secrets and creating a secret from these certificates for use within the cluster. These certificates are automatically available to new nodes joining the cluster.

Follow the below steps to provide the etcd certificates to Portworx using Kubernetes Secrets:

  1. Copy all your etcd certificates and key into a directory /kvdb-secrets to create a Kubernetes secret from it. Ensure that the file names match the default names given below:
ls -1 kvdb-secrets/
kvdb-ca.crt
kvdb.crt
kvdb.key
  1. Create the secret named px-kvdb-auth from the above files:
kubectl -n <px-namespace> create secret generic px-kvdb-auth \
--from-file=kvdb-secrets/
secret/px-kvdb-auth created
  1. Verify that the secret has three keys: kvdb-ca.crt, kvdb.crt, and kvdb.key, corresponding to the filenames in the kvdb-secrets folder. These keys are used by Portworx to reference the certificates:
kubectl -n <px-namespace> describe secret px-kvdb-auth
Name:         px-kvdb-auth
Namespace:    <px-namespace>
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
kvdb-ca.crt:      1679 bytes
kvdb.crt:  1680 bytes
kvdb.key:  414  bytes

Reference keys for etcd integration with Portworx​

To enhance the security of your Portworx cluster and seamlessly integrate it with an etcd cluster, follow these steps to reference the keys created to secure the etcd cluster:

  1. Log in to Portworx Central to generate a Portworx spec.
  2. Click the Customize button at the bottom of the page.
  3. Select Your etcd details on the Basic window to input your etcd cluster endpoints, and choose the Certificate Auth option to secure your external etcd cluster.
  4. Follow the wizard to generate the Portworx spec.

Verifying external etcd endpoints in the Portworx spec​

After generating the spec, confirm that the kvdb section of the StorageCluster resource includes your etcd endpoints and authentication secret:

spec:
  kvdb:
    endpoints:
      - etcd:https://<your-etcd-endpoint1>:2379
      - etcd:https://<your-etcd-endpoint2>:2379
      - etcd:https://<your-etcd-endpoint3>:2379
    authSecret: px-kvdb-auth

When you install Portworx using the generated spec, it uses your specified cluster as an external KVDB, ensuring secure communication. For more information on how to generate a spec for your environment, see Install Portworx Enterprise.