External KVDB for Portworx in airgapped EKS

Summary and Key concepts

Summary:

This article provides guidelines for configuring and using an external KVDB with Portworx, specifically utilizing an etcd cluster as the key-value store. It outlines the prerequisites for setting up a 3-node etcd cluster, including memory, drive size, and IOPS requirements. The article covers best practices for maintaining the etcd cluster, such as setting up regular compaction, space quotas, and snapshots for recovery. It also describes how to secure communication between Portworx and the etcd cluster using Kubernetes Secrets, and how to integrate the etcd cluster into Portworx by referencing etcd endpoints and authentication keys in the Portworx spec.

Kubernetes Concepts:

• Kubernetes Secrets: Used to store and manage sensitive information like SSL certificates for securing etcd communication.

Portworx Concepts:

• Run-flat mode: Implications on External KVBD.

Portworx can use an external KVDB to store its metadata and configuration data.

etcd is the only external KVDB supported by Portworx, which is a highly available clustered database that can be scaled independently. Therefore, follow the instructions on this page to configure an etcd cluster as your external KVDB.

Prerequisites

• A 3-node etcd cluster with etcd version 3.3 or newer is required.

• Allocate a minimum of 8 GB of memory dedicated to each etcd node.

• KVDB drive:

  • If IOPS are independent of disk size, the minimum recommended size is 32 GB or a minimum of 450 IOPs.

  • If IOPS are dependent on disk size, the recommended size is 150 GB.

• Ensure that the recommended hardware requirements for your environment are met.

Configure your etcd cluster

Follow the following best practices to keep your etcd cluster up and healthy.

Compaction

etcd stores a complete history of its keyspace, and regular compaction is necessary to avoid performance issues and running out of storage space. Regular compaction helps manage the memory usage of the etcd process.

You can configure the compaction period using the etcd tool with the --auto-compaction-retention option.

For Portworx, it is recommended to retain the history for the last 3 hours. To achieve this, set the --auto-compaction-retention value to 3. For information on how to set the compaction period, see the etcd documentation.

Space Quota

A space quota, or setting a database size limit in etcd, is crucial for maintaining a stable cluster. Without a space quota, etcd can experience performance issues when the keyspace grows excessively and the available storage space is consumed, resulting in unpredictable cluster behavior.

For Portworx, it is recommended to set the space quota to a maximum value of 8 GiB. To achieve this, use the etcd tool to set the --quota-backend-bytes to $((8*1024*1024*1024))

For more information on how to set the space quota, see the etcd documentation.

Snapshots

You can recover your etcd cluster in the event of a disaster by taking a snapshot of its keyspace. Run the following command as part of a cron job, either on the etcd nodes or on a separate node where you want to store these periodic etcd snapshots:

ETCDCTL_API=3 etcdctl --endpoints="<comma-separated-etcd-url>" snapshot save </path/to/snapshot-file> --command-timeout=60s

The above example command will store etcd snapshots to the specified file and if any etcd operation takes longer than 60 seconds, it will time out and be canceled.

For more information on how to setup a recovery mechanism, see the etcd documentation.

Secure your etcd communication

You can configure and secure etcd communication in your environment by storing SSL certificates as Kubernetes secrets and creating a secret from these certificates for use within the cluster. These certificates will automatically be available to new nodes joining the cluster.

Follow the below steps to provide the etcd certificates to Portworx using Kubernetes Secrets:

1. Copy all your etcd certificates and key into a directory /kvdb-secrets to create a Kubernetes secret from it. Ensure that the file names match the default names given below:

ls -1 kvdb-secrets/

kvdb-ca.crt
kvdb.crt
kvdb.key

2. Use the following command to create the secret named px-kvdb-auth from the above files:

kubectl -n <px-namespace> create secret generic px-kvdb-auth \
--from-file=kvdb-secrets/

secret/px-kvdb-auth created

3. Verify that the secret has three keys: kvdb-ca.crt, kvdb.crt, and kvdb.key, corresponding to the filenames in the kvdb-secrets folder. These keys are used by Portworx to reference the certificates:

kubectl -n <px-namespace> describe secret px-kvdb-auth

Name:         px-kvdb-auth
Namespace:    <px-namespace>
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
kvdb-ca.crt:      1679 bytes
kvdb.crt:  1680 bytes
kvdb.key:  414  bytes

Reference keys for etcd integration with Portworx

To enhance the security of your Portworx cluster and seamlessly integrate it with an etcd cluster, follow these steps to reference the keys created to secure the etcd cluster:

1. Navigate to Portworx Central to generate a Portworx spec.
2. Click the Customize button at the bottom of the page.
3. Select Your etcd details on the Basic window to input your etcd cluster endpoints, and choose the Certificate Auth option to secure your external etcd cluster.
4. Follow the wizard to generate the Portworx spec.

Verify external etcd endpoints in the Portworx spec

Once the spec is generated, you will find the following entries in the kvdb section of your Portworx StorageCluster. When you install Portworx using the generated spec, it will use your specified cluster as an external KVDB, ensuring secure communication:

spec:
  kvdb:
    endpoints:
      - etcd:https://<your-etcd-endpoint1>:2379
      - etcd:https://<your-etcd-endpoint2>:2379
      - etcd:https://<your-etcd-endpoint3>:2379
    authSecret: px-kvdb-auth

For more information on how to generate a spec for your environment, see the install documentation.