Skip to main content
Version: 3.5

Vault

Portworx can integrate with Vault to store your encryption keys, secrets, and credentials. This topic explains how to connect a Portworx cluster to a Vault development server endpoint and use it to store secrets for encrypting volumes, as well as credentials such as vSphere credentials.

You can either use Vault for volume encryption and Kubernetes Secrets for storing vSphere credentials, or use Vault for both purposes.

Set up Vault​

Set up and deploy Vault by following the instructions in the Install Vault section of the Vault documentation. This includes installation, setting up policies, and configuring secrets.

note

To run a dev server, use the vault server -dev command. This will only run on 127.0.0.1:8200, and cannot be connected by the container. Ensure the server endpoint is securely exposed to the Portworx clusters.

## Authenticate Vault with Portworx​

Step 1: Choose the Vault authentication method.​

Authentication methods are responsible for authenticating Portworx with Vault. Based on your Vault configuration and the authentication method you choose, you must use one of the following two methods:

  • Using Token authentication: A static Vault token is provided to Portworx.
  • Using Kubernetes authentication: Portworx uses Kubernetes service account to fetch and refresh Vault tokens.
  • Using Vault AppRole authentication: Portworx uses Vault AppRole's Role ID and Secret ID to authenticate and generate Vault Tokens.

Using token authentication method​

With this method, Portworx requires a Vault static token that you should provide through a Kubernetes secret.

Provide Vault credentials to Portworx. Refer to the Vault credentials reference for details on the credentials.

Create the Kubernetes secret with the name px-vault in the portworx namespace. If PX_SECRETS_NAMESPACE is set, create the secret in the defined namespace. For example

apiVersion: v1
kind: Secret
metadata:
  name: px-vault
  namespace: portworx
type: Opaque
data:
  VAULT_ADDR: (required)<base64 encoded value of the vault endpoint address>
  VAULT_TOKEN: (required)<base64 encoded value of the vault token>
  VAULT_CACERT: (recommended)<base64 encoded file path where the CA Certificate is present on all the nodes>
  VAULT_CAPATH: (recommended)<base64 encoded file path where the Certificate Authority is present on all the nodes>
  VAULT_CLIENT_CERT: (recommended)<base64 encoded file path where the Client Certificate is present on all the nodes>
  VAULT_CLIENT_KEY: (recommended)<base64 encoded file path where the Client Key is present on all the nodes>
  VAULT_TLS_SERVER_NAME: (recommended)<base64 encoded value of the TLS server name>
  VAULT_BACKEND_PATH: (optional)<base64 encoded value of the custom backend path if different than the default "secret">
  VAULT_NAMESPACE: (optional)<base64 encoded value of the global vault namespace for portworx>

Portworx searches for this secret with name px-vault under the portworx namespace.

note

If the VAULT_TOKEN provided in the secret above is refreshed, then you must manually update this secret.

After configuring Vault using the Vault authentication method, proceed to Step 2.

Using Kubernetes authentication method​

This method allows Portworx to authenticate with Vault using a Kubernetes service account token. For more information about how to set up Kubernetes Vault authentication, refer to the Vault documentation.

  1. Create a ServiceAccount for Vault authentication delegation.

    Run the following kubectl create or oc create commands to create a ServiceAccount and ClusterRoleBinding. Vault uses this ServiceAccount and its associated token to authenticate requests from Portworx. Vault uses the Kubernetes TokenReview API.

    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: vault-auth
      namespace: portworx
    secrets:
      - name: vault-auth-token
    EOF
    serviceaccount/vault-auth created
    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: Secret
    metadata:
      name: vault-auth-token
      namespace: portworx
      annotations:
        kubernetes.io/service-account.name: vault-auth
    type: kubernetes.io/service-account-token
    EOF
    secret/vault-auth-token created
    kubectl create clusterrolebinding vault-tokenreview-binding --clusterrole=system:auth-delegator --serviceaccount=portworx:vault-auth
    clusterrolebinding.rbac.authorization.k8s.io/vault-tokenreview-binding created

  2. Enable Kubernetes authentication in Vault. Enter the following vault auth command to enable Kubernetes authentication in Vault:

    vault auth enable kubernetes

  3. Create a Kubernetes authentication configuration in Vault. Enter the following export commands to get the JWT token and CA certificate of Kubernetes ServiceAccount:

export SA_JWT_TOKEN=$(kubectl get secret vault-auth-token -n portworx \
  -o jsonpath="{.data.token}" | base64 --decode; echo)

export SA_CA_CRT=$(kubectl get secret vault-auth-token -n portworx \
  -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)

Enter the following vault write command, replacing <kubernetes-endpoint> with your Kubernetes API-server endpoint to write a Kubernetes authentication configuration to Vault:

vault write auth/kubernetes/config \
token_reviewer_jwt="$SA_JWT_TOKEN" \
kubernetes_host="<kubernetes endpoint>" \
kubernetes_ca_cert="$SA_CA_CRT" \
issuer="https://kubernetes.default.svc.cluster.local" # Optional

  1. Create a Kubernetes authentication role for Portworx, named portworx, in Vault:

    vault write auth/kubernetes/role/portworx \
      bound_service_account_names=portworx \
      bound_service_account_namespaces=<namespace> \
      policies=portworx \
      ttl=<ttl>

  2. Provide Vault credentials to Portworx. Refer to Vault credentials reference for details on the credentials.

    Portworx reads the Vault credentials required to authenticate with Vault through a Kubernetes secret. Create the Kubernetes secret in the namespace where Portworx is deployed, for example portworx or portworx. If PX_SECRETS_NAMESPACE is set, create the secret in the defined namespace. For example:

    apiVersion: v1
    kind: Secret
    metadata:
      name: px-vault
      namespace: portworx
    type: Opaque
    data:
      VAULT_ADDR: <base64 encoded value of the vault endpoint address>
      VAULT_BACKEND_PATH: <base64 encoded value of the custom backend path if different than the default "secret">
      VAULT_CACERT: <base64 encoded file path where the CA Certificate is present on all the nodes>
      VAULT_CAPATH: <base64 encoded file path where the Certificate Authority is present on all the nodes>
      VAULT_CLIENT_CERT: <base64 encoded file path where the Client Certificate is present on all the nodes>
      VAULT_CLIENT_KEY: <base64 encoded file path where the Client Key is present on all the nodes>
      VAULT_TLS_SERVER_NAME: <base64 encoded value of the TLS server name>
      VAULT_AUTH_METHOD: a3ViZXJuZXRlcw== # base64 encoded value of "kubernetes"
      VAULT_AUTH_KUBERNETES_ROLE: cG9ydHdvcng= # base64 encoded value of the kubernetes auth role "portworx"
      VAULT_NAMESPACE: <base64 encoded value of the global vault namespace for portworx>

During installation, Portworx creates a Kubernetes role binding that grants read access to Kubernetes secrets from only the defined namespace.

Using AppRole authentication method​

This method allows Portworx to authenticate with Vault using AppRole authentication. AppRole authentication requires a Role ID and a Secret ID. For more information about how to set up AppRole authentication, refer to the Vault AppRole documentation.

Setup AppRole in Vault​
  1. Enable AppRole in Vault using the following command:
vault auth enable approle

  1. Create a policy that will be used by Vault tokens. Use the policies defined in Vault security policies, and store the policy in portworx.hcl:

    vault policy write portworx portworx.hcl

  2. Create a role named my-role with the token policy my-policies using a command similar to the following:

    vault write auth/approle/role/my-role \
      token_num_uses=10 \
      token_ttl=20m \
      token_max_ttl=30m  \
      token_policies=my-policy

token_num_uses, token_ttl, token_max_ttl, and token_policies are restrictions on how the generated token can be used to log in to Vault using AppRole.

note

There are two additional parameters, secret_id_ttl and secret_id_num_uses, that you might see in the reference. Portworx by Everpure recommends setting secret_id_ttl and secret_id_num_uses to 0. If the Secret ID is expired, you need to update VAULT_APPROLE_SECRET_ID in either the Kubernetes secret or the environment variable as specified in the following section.

  1. Obtain the Role ID and Secret ID of my-role for authentication using the following commands:

    vault read auth/approle/role/my-role/role-id
    role_id     xxxxxxxx-xxxx-xxxx-xxxx-67221c5c2f63
    vault write -f auth/approle/role/my-role/secret-id
    secret_id               xxxxxxxx-xxxx-xxxx-xxxx-6018fcceff64
    secret_id_accessor      xxxxxxxx-xxxx-xxxx-xxxx-6ef26b7bcf86
Provide Vault AppRole credentials to Portworx​

Portworx reads the Vault credentials required to authenticate with Vault through a Kubernetes secret.

Create the Kubernetes secret with the name px-vault in the portworx namespace. In the case PX_SECRETS_NAMESPACE is set, create the secret in the defined namespace.

apiVersion: v1
kind: Secret
metadata:
  name: px-vault
  namespace: portworx
type: Opaque
data:
  VAULT_ADDR: <base64 encoded value of the vault endpoint address>
  VAULT_BACKEND_PATH: <base64 encoded value of the custom backend path if different than the default "secret">
  VAULT_CACERT: <base64 encoded file path where the CA Certificate is present on all the nodes>
  VAULT_CAPATH: <base64 encoded file path where the Certificate Authority is present on all the nodes>
  VAULT_CLIENT_CERT: <base64 encoded file path where the Client Certificate is present on all the nodes>
  VAULT_CLIENT_KEY: <base64 encoded file path where the Client Key is present on all the nodes>
  VAULT_TLS_SERVER_NAME: <base64 encoded value of the TLS server name>
  VAULT_AUTH_METHOD: YXBwcm9sZQ== # base64 encoded value of "approle"
  VAULT_APPROLE_ROLE_ID: <base64 encoded value of the Role ID>
  VAULT_APPROLE_SECRET_ID: <base64 encoded value of the Secret ID>
  VAULT_NAMESPACE: <base64 encoded value of the global vault namespace for portworx>

For AppRole authentication, the three additioanl parameters are VAULT_AUTH_METHOD, VAULT_APPROLE_ROLE_ID, and VAULT_APPROLE_SECRET_ID.

Step 2: Setup Vault as the secrets provider for Portworx​

New Installation​

When generating the Portworx Kubernetes specification file, select Vault from the "Secrets type" list.

Existing Installation​

Edit your StorageCluster object, setting the value of the specs.secretsProvider field to vault.

Updating the secrets provider

If you update specs.secretsProvider from k8s to vault and have existing volumes that are encrypted using the Kubernetes secrets key, store the same secret key in Vault before the transition.

For example, if a Kubernetes secret has the name <your-secret-name> and the key <your-secret-key> with the value <your-secret-passcode>, create a Vault secret with the same name and key before transitioning to Vault as the secrets provider.

The following example shows how to create a Kubernetes secret when using the Kubernetes secrets provider:

kubectl create secret generic <your-secret-name> -n portworx --from-literal=your-secret-key=<your-secret-passcode>

Create a Vault secret with the same name and key, as shown in the example below. Here, <customPath> is a custom backend path provided in the px-vault secret see Vault credentials reference for more information:

vault kv put <customPath>/<your-secret-name> <your-secret-key>=<your-secret-passcode>

Vault security policies​

If you configured Vault strictly with policies, then the Vault token provided to Portworx should follow one of the following policies:

# Read and List capabilities on mount to determine which version of kv backend is supported
path "sys/mounts/"
{
capabilities = ["read", "list"]
}

# V1 backends (Using default backend)
# Provide full access to the portworx subkey
path "secret/*"
{
capabilities = ["create", "read", "update", "delete", "list"]
}

# V1 backends (Using custom backend)
# Provide full access to the portworx subkey
# Provide -> VAULT_BACKEND_PATH=custom-backend (required)
path "custom-backend/*"
{
capabilities = ["create", "read", "update", "delete", "list"]
}


# V2 backends (Using default backend )
# Provide full access to the data/portworx subkey
path "secret/data/*"
{
capabilities = ["create", "read", "update", "delete", "list"]
}

# V2 backends (Using custom backend )
# Provide full access to the data/portworx subkey
# Provide -> VAULT_BACKEND_PATH=custom-backend (required)
path "custom-backend/data/*"
{
capabilities = ["create", "read", "update", "delete", "list"]
}
note

Portworx supports only the kv backend of Vault.

You can set all the above Vault related fields and the cluster secret key using the Portworx CLI, which is explained in the next section.

Run the following command to create a policy that by using the above policies

vault policy write portworx portworx.hcl

Key generation with Vault​

vSphere Credentials for Portworx​

When using Portworx with vSphere, you can supply the vSphere credentials by creating a secret in Vault. Follow the steps below to properly create and verify the secret.

warning

If you were using Kubernetes Secret to store vSphere credentials, make sure you don't have a secret in your cluster named px-vsphere-secret, and that the StorageCluster spec doesn't have the VSPHERE_USER and VSPHERE_PASSWORD environment variables set.

To delete environment variables from the StorageCluster, run the following commands:

## Get the StorageCluster name
kubectl get storagecluster -n <portworx>
## Edit the StorageCluster
kubectl edit storagecluster -n <portworx> <stc_name> ## replace <stc_name> with the name of your StorageCluster

In the env section, remove the following lines:

env:
  - name: VSPHERE_INSECURE
    value: "true"
  - name: VSPHERE_USER
    valueFrom:
      secretKeyRef:
        name: px-vsphere-secret
        key: VSPHERE_USER
  - name: VSPHERE_PASSWORD
    valueFrom:
      secretKeyRef:
        name: px-vsphere-secret
        key: VSPHERE_PASSWORD

Otherwise, those settings will take precedence over the Vault credential store. As a result, Portworx will not use the Vault credentials for vSphere credentials and will instead rely on the Kubernetes secret store.

Step 1: Create the Secret​

Store vSphere credentials (username and password) as key-value pairs at the path secret/vsphereCredentials. If you are not using a custom backend path, use secret as shown. If you are using a custom secret path, replace secret with your custom path.

Run the following command, substituting the placeholder values with your actual credentials:

vault kv put secret/vsphereCredentials VSPHERE_PASSWORD=<vsphere_password> VSPHERE_USER=<vsphere_user>

If your custom secret path is, for example, customPath:

vault kv put customPath/vsphereCredentials VSPHERE_PASSWORD=<vsphere_password> VSPHERE_USER=<vsphere_user>

Step 2: Retrieve and Verify the Secret​

To ensure that the secret was created correctly, use this command:

vault kv get secret/vsphereCredentials

The command should return an output similar to the following:

========== Data ==========
Key                 Value
---                 -----
VSPHERE_PASSWORD    <vsphere_password>
VSPHERE_USER        <vsphere_user>

Volume encryption​

The following sections describe the key generation process with Portworx and Vault which you can use for encrypting volumes. For more information about encrypted volumes, refer to the Encrypted volumes using pxctl page.

Step 1: Store the secret in Vault​

vault kv put secret/px-vault foo=bar

Step 2: Set a cluster wide secret​

A cluster wide secret key is a common key that can be used to encrypt all your volumes. This common key needs to be pre-created in your KMS provider.

You can set the cluster secret key using the following command:

pxctl secrets set-cluster-key
Enter cluster wide secret key: *****
Successfully set cluster secret key!

or

Provide it by a flag --secret.

pxctl secrets set-cluster-key --secret <cluster-wide-secret-key>

In the above prompt you need to enter the secret key that you created in your KMS. You should run this command only once for the cluster. If you added the cluster secret key through the config.json, then the command above overwrites it. Even on subsequent Portworx restarts, the cluster secret key in config.json will be ignored for the one set through the CLI.

note

To use the secret we just created, enter px-vault for the the cluster wide secret.

Vault credentials reference​

important

The file paths specified for VAULT_CACERT, VAULT_CAPATH, VAULT_CLIENT_CERT, and VAULT_CLIENT_KEY must reside in directories accessible to the Portworx pods through HostPath volume mounts. Portworx by Everpure recommends using /opt/pwx or /etc/pwx as file paths.

Portworx requires the following Vault credentials to use its APIs:

  • Vault Address [VAULT_ADDR]

    Vault server address expressed as a URL and port. For example: https://192.168.11.11:8200

  • Vault Token [VAULT_TOKEN]

    Vault authentication token. Refer to the Vault tokens page for more information about Vault tokens. If you are using the Kubernetes authentication method of Vault, then you need not provide the actual token to Portworx.

  • Vault Backend Path [VAULT_BACKEND_PATH]

    The custom backend path if different than the default secret

  • Vault CA Certificate [VAULT_CACERT]

    Path to a PEM-encoded CA certificate file that needs to be present on all Portworx nodes. This file is used to verify the SSL certificate of Vault server. This variable takes precedence over VAULT_CAPATH.

  • Vault CA Path [VAULT_CAPATH]

    Path to a directory of PEM-encoded CA certificate files that needs to be present on all Portworx nodes.

  • Vault Client Certificate [VAULT_CLIENT_CERT]

    Path to a PEM-encoded client certificate that needs to be present on all Portworx nodes. This file is used for TLS communication with the Vault server.

  • Vault Client Key [VAULT_CLIENT_KEY]

    Path to an unencrypted, PEM-encoded private key which corresponds to the matching client certificate. This key file needs to be present on all Portworx nodes.

  • Vault TLS Server Name [VAULT_TLS_SERVER_NAME]

    Name to use as the SNI host when you connect using TLS.

  • Vault Auth Method [VAULT_AUTH_METHOD]

    Specifies the authentication method that Portworx should use while authenticating with Vault. "Kubernetes" is the currently supported authentication method.

  • Vault Auth Kubernetes Role [VAULT_AUTH_KUBERNETES_ROLE]

    Name of the Kubernetes "Auth Role" created in Vault for Portworx. This field is set only when using the Kubernetes authentication method.

  • Vault Namespace [VAULT_NAMESPACE]

    Allows you to set a global Vault namespace for the Portworx cluster. All Vault requests by Portworx use this Vault namespace, if you do not provide an override.

Using Vault with Portworx​

Last updated on