External KVDB for Portworx on Kubernetes

Portworx can use an external KVDB to store its metadata and configuration data.

etcd is the only external KVDB supported by Portworx, which is a highly available clustered database that can be scaled independently. Therefore, follow the instructions on this page to configure an etcd cluster as your external KVDB.

Prerequisites

• A 3-node etcd cluster with etcd version 3.3 or newer is required.
• Allocate a minimum of 8 GB of memory dedicated to each etcd node.
• KVDB drive:
  • If IOPS are independent of disk size, the minimum recommended size is 32 GB or a minimum of 450 IOPs.
  • If IOPS are dependent on disk size, the recommended size is 150 GB.
    note

    If you are using a cloud drive, you should size it in accordance with your cloud provider's specifications.

• Ensure that the recommended hardware requirements for your environment are met.

Configure your etcd cluster

Follow the following best practices to keep your etcd cluster up and healthy.

Compaction

etcd stores a complete history of its keyspace, and regular compaction is necessary to avoid performance issues and running out of storage space. Regular compaction helps manage the memory usage of the etcd process.

You can configure the compaction period using the etcd tool with the --auto-compaction-retention option.

For Portworx, it is recommended to retain the history for the last 3 hours. To achieve this, set the --auto-compaction-retention value to 3. For information on how to set the compaction period, see the etcd documentation.

Space Quota

A space quota, or setting a database size limit in etcd, is crucial for maintaining a stable cluster. Without a space quota, etcd can experience performance issues when the keyspace grows excessively and the available storage space is consumed, resulting in unpredictable cluster behavior.

For Portworx, it is recommended to set the space quota to a maximum value of 8 GiB. To achieve this, use the etcd tool to set the --quota-backend-bytes to $((8*1024*1024*1024))

For more information on how to set the space quota, see the etcd documentation.

Snapshots

You can recover your etcd cluster in the event of a disaster by taking a snapshot of its keyspace. Run the following command as part of a cron job, either on the etcd nodes or on a separate node where you want to store these periodic etcd snapshots:

ETCDCTL_API=3 etcdctl --endpoints="<comma-separated-etcd-url>" snapshot save </path/to/snapshot-file> --command-timeout=60s

The above example command will store etcd snapshots to the specified file and if any etcd operation takes longer than 60 seconds, it will time out and be canceled.

For more information on how to setup a recovery mechanism, see the etcd documentation.

Secure your etcd communication

You can configure and secure etcd communication in a Kubernetes environment by storing SSL certificates as Kubernetes secrets and creating a secret from these certificates for use within the cluster. These certificates will automatically be available to new nodes joining the cluster.

Follow the below steps to provide the etcd certificates to Portworx using Kubernetes Secrets:

1. Copy all your etcd certificates and key into a directory /kvdb-secrets to create a Kubernetes secret from it. Ensure that the file names match the default names given below:

   ls -1 kvdb-secrets/

   kvdb-ca.crt
   kvdb.crt
   kvdb.key

2. Use the following command to create the secret named px-kvdb-auth from the above files:

   kubectl -n <px-namespace> create secret generic px-kvdb-auth \
   --from-file=kvdb-secrets/

   secret/px-kvdb-auth created

3. Verify that the secret has three keys: kvdb-ca.crt, kvdb.crt, and kvdb.key, corresponding to the filenames in the kvdb-secrets folder. These keys are used by Portworx to reference the certificates:

   kubectl -n <px-namespace> describe secret px-kvdb-auth

   Name:         px-kvdb-auth
   Namespace:    <px-namespace>
   Labels:       <none>
   Annotations:  <none>
   
   Type:  Opaque
   
   Data
   ====
   kvdb-ca.crt:      1679 bytes
   kvdb.crt:  1680 bytes
   kvdb.key:  414  bytes

Reference keys for etcd integration with Portworx

To enhance the security of your Portworx cluster and seamlessly integrate it with an etcd cluster, follow these steps to reference the keys created to secure the etcd cluster:

1. Navigate to Portworx Central to generate a Portworx spec.
2. Click the Customize button at the bottom of the page.
3. Select Your etcd details on the Basic window to input your etcd cluster endpoints, and choose the Certificate Auth option to secure your external etcd cluster.
4. Follow the wizard to generate the Portworx spec.

Verify external etcd endpoints in the Portworx spec

Once the spec is generated, you will find the following entries in the kvdb section of your Portworx StorageCluster. When you install Portworx using the generated spec, it will use your specified cluster as an external KVDB, ensuring secure communication:

spec:
  kvdb:
    endpoints:
    - etcd:https://<your-etcd-endpoint1>:2379
    - etcd:https://<your-etcd-endpoint2>:2379
    - etcd:https://<your-etcd-endpoint3>:2379
    authSecret: px-kvdb-auth

For more information on how to generate a spec for your environment, see the install documentation.