Vault in OCP GCP
Portworx can integrate with Vault to store your encryption keys, secrets, and credentials. This topic explains how to connect a Portworx cluster to a Vault development server endpoint and use it to store secrets that you can use for encrypting volumes.
Set up Vault
Set up and deploy Vault by following the instructions in the Install Vault section of the Vault documentation. This includes installation, setting up policies, and configuring secrets.
To run a dev server, use the vault server -dev
command. This will only run on 127.0.0.1:8200, and cannot be connected by the container. Ensure the server endpoint is securely exposed to the Portworx clusters.
Set up the Vault development environment
Once you've set up Vault, you're ready to set up your development environment.
Pure Storage does not recommend using this for production environments.
-
Create a
config.hcl
file, then start the Vault server:config.hcl
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 1
}
storage "file" {
path = "/tmp/vault-data"
}
disable_mlock = truemkdir -p /tmp/vault-data
vault server -config=config.hcl -
When Vault initializes, it will present the unseal keys and initial root token. Securely store and distribute the keys, as they will be used in later operations.
export VAULT_ADDR='http://127.0.0.1:8200'
vault operator initUnseal Key 1: 4jYbl2CBIv6SpkKj6Hos9iD32k5RfGkLzlosrrq/JgOm
Unseal Key 2: B05G1DRtfYckFV5BbdBvXq0wkK5HFqB9g2jcDmNfTQiS
Unseal Key 3: Arig0N9rN9ezkTRo7qTB7gsIZDaonOcc53EHo83F5chA
Unseal Key 4: 0cZE0C/gEk3YHaKjIWxhyyfs8REhqkRW/CSXTnmTilv+
Unseal Key 5: fYhZOseRgzxmJCmIqUdxEm9C3jB5Q27AowER9w4FC2Ck
Initial Root Token: <your-root-token>
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated main key. Without at least 3 key to
reconstruct the main key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information. -
When you first start Vault, you must unseal it. Enter the following command to unseal the Vault server. Repeat it 3 times:
vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
Unseal Nonce xxxxxxxx-xxxx-xxxx-xxxx-e63ddb34b2a9
Version 1.7.0
Storage Type raft
HA Enabled true -
Log in to the Vault server using the root token you generated in step 2:
vault login <initial-root-token>
-
Verify the installation by entering the following vault command. Specify your own value for
<my-vault-secret>
:vault kv put secret/my-secret my-value=<my-vault-secret>
Key Value
--- -----
created_time 2019-06-19T17:20:22.985303Z
deletion_time n/a
destroyed false
version 1
Kubernetes users
Step 1: Choose the Vault authentication method.
Authentication methods are responsible for authenticating Portworx with Vault. Based on your Vault configuration and the authentication method you choose, you must use one of the following two methods:
- Using Token authetication: A static Vault token is provided to Portworx.
- Using Kubernetes authentication: Portworx uses Kubernetes service account to fetch and refresh Vault tokens.
- Using Vault AppRole authentication: Portworx uses Vault AppRole's Role ID and Secret ID to authenticate and generate Vault Tokens.
Using token authentication method
With this method, Portworx requires a Vault static token that you should provide through a Kubernetes secret.
Provide Vault credentials to Portworx. Refer to the Vault credentials reference for details on the credentials.
Create the Kubernetes secret with the name px-vault
in the kube-system
namespace. If PX_SECRETS_NAMESPACE
is set, create the secret in the defined namespace.
For example
apiVersion: v1
kind: Secret
metadata:
name: px-vault
namespace: portworx
type: Opaque
data:
VAULT_ADDR: (required)<base64 encoded value of the vault endpoint address>
VAULT_TOKEN: (required)<base64 encoded value of the vault token>
VAULT_CACERT: (recommended)<base64 encoded file path where the CA Certificate is present on all the nodes>
VAULT_CAPATH: (recommended)<base64 encoded file path where the Certificate Authority is present on all the nodes>
VAULT_CLIENT_CERT: (recommended)<base64 encoded file path where the Client Certificate is present on all the nodes>
VAULT_CLIENT_KEY: (recommended)<base64 encoded file path where the Client Key is present on all the nodes>
VAULT_TLS_SERVER_NAME: (recommended)<base64 encoded value of the TLS server name>
VAULT_BACKEND_PATH: (optional)<base64 encoded value of the custom backend path if different than the default "secret">
VAULT_NAMESPACE: (optional)<base64 encoded value of the global vault namespace for portworx>
Portworx searches for this secret with name px-vault
under the portworx
namespace.
If the VAULT_TOKEN
provided in the secret above is refreshed, then you must manually update this secret.
After configuring Vault using the Vault authentication method, proceed to Step 2.
Using Kubernetes authentication method
This method allows Portworx to authenticate with Vault using a Kubernetes service account token. For more information about how to setup Kubernetes Vault authentication, refer to the Vault documentation.
-
Create a
ServiceAccount
for Vault authentication delegation.Run the following
oc create
commands to create aServiceAccount
andClusterRoleBinding
. Vault uses thisServiceAccount
and its associated token to authenticate requests from Portworx. Vault uses the Kubernetes TokenReview API.oc create serviceaccount vault-auth -n kube-system
oc create clusterrolebinding vault-tokenreview-binding --clusterrole=system:auth-delegator --serviceaccount=kube-system:vault-auth -
Enable Kubernetes authentication in Vault. Enter the following
vault auth
command to enable Kubernetes authentication in Vault:vault auth enable kubernetes
-
Create a Kubernetes authentication configuration in Vault. Enter the following export commands to get the JWT token and CA certificate of Kubernetes ServiceAccount:
export VAULT_SA_NAME=$(oc get sa vault-auth -n kube-system \
-o jsonpath="{.secrets[*]['name']}")
export SA_JWT_TOKEN=$(oc get secret $VAULT_SA_NAME -n kube-system \
-o jsonpath="{.data.token}" | base64 --decode; echo)
export SA_CA_CRT=$(oc get secret $VAULT_SA_NAME -n kube-system \
-o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)
Enter the following vault write command, replacing <kubernetes-endpoint>
with your Kubernetes API-server endpoint to write a Kubernetes authentication configuration to Vault:
vault write auth/kubernetes/config \
token_reviewer_jwt="$SA_JWT_TOKEN" \
kubernetes_host="<kubernetes endpoint" \
kubernetes_ca_cert="$SA_CA_CRT" \
issuer="https://kubernetes.default.svc.cluster.local"
-
Create a Kubernetes authentication role for Portworx, named
portworx
, in Vault:vault write auth/kubernetes/role/portworx \
bound_service_account_names=portworx \
bound_service_account_namespaces=<namespace> \
policies=portworx \
ttl=<ttl> -
Provide Vault credentials to Portworx. Refer to Vault credentials reference for details on the credentials.
Portworx reads the Vault credentials required to authenticate with Vault through a Kubernetes secret. Create the Kubernetes secret in the namespace where Portworx is deployed, for example
portworx
orkube-system
. IfPX_SECRETS_NAMESPACE
is set, create the secret in the defined namespace. For example:apiVersion: v1
kind: Secret
metadata:
name: px-vault
namespace: portworx
type: Opaque
data:
VAULT_ADDR: <base64 encoded value of the vault endpoint address>
VAULT_BACKEND_PATH: <base64 encoded value of the custom backend path if different than the default "secret">
VAULT_CACERT: <base64 encoded file path where the CA Certificate is present on all the nodes>
VAULT_CAPATH: <base64 encoded file path where the Certificate Authority is present on all the nodes>
VAULT_CLIENT_CERT: <base64 encoded file path where the Client Certificate is present on all the nodes>
VAULT_CLIENT_KEY: <base64 encoded file path where the Client Key is present on all the nodes>
VAULT_TLS_SERVER_NAME: <base64 encoded value of the TLS server name>
VAULT_AUTH_METHOD: a3ViZXJuZXRlcw== # base64 encoded value of "kubernetes"
VAULT_AUTH_KUBERNETES_ROLE: cG9ydHdvcng= # base64 encoded value of the kubernetes auth role "portworx"
VAULT_NAMESPACE: <base64 encoded value of the global vault namespace for portworx>
During installation, Portworx creates a Kubernetes role binding that grants read access to Kubernetes secrets from only the defined namespace.
Using AppRole authentication method
This method allows Portworx to authenticate with Vault using AppRole
authentication. AppRole
authentication requires a Role ID and a Secret ID. For more information about how to set up AppRole authentication, refer to the Vault AppRole documentation.
Setup AppRole in Vault
- Enable AppRole in Vault using the following command:
vault auth enable approle
-
Create a policy that will be used by Vault tokens. Use the policies defined in Vault security policies, and store the policy in portworx.hcl:
vault policy write portworx portworx.hcl
-
Create a role named
my-role
with the token policymy-policies
using a command similar to the following:vault write auth/approle/role/my-role \
token_num_uses=10 \
token_ttl=20m \
token_max_ttl=30m \
token_policies=my-policy
token_num_uses
, token_ttl
, token_max_ttl
, and token_policies
are restrictions on how the generated token can be used to log in to Vault using AppRole.
There are two additional parameters, secret_id_ttl
and secret_id_num_uses
, that you might see in the reference. Portworx by Pure Storage recommends setting secret_id_ttl
and secret_id_num_uses
to 0
. If the Secret ID is expired, you need to update VAULT_APPROLE_SECRET_ID
in either the Kubernetes secret or the environment variable as specified in the following section.
-
Obtain the Role ID and Secret ID of
my-role
for authentication using the following commands:vault read auth/approle/role/my-role/role-id
role_id xxxxxxxx-xxxx-xxxx-xxxx-67221c5c2f63
vault write -f auth/approle/role/my-role/secret-id
secret_id xxxxxxxx-xxxx-xxxx-xxxx-6018fcceff64
secret_id_accessor xxxxxxxx-xxxx-xxxx-xxxx-6ef26b7bcf86
Provide Vault AppRole credentials to Portworx
Portworx reads the Vault credentials required to authenticate with Vault through a Kubernetes secret.
Create the Kubernetes secret with the name px-vault
in the kube-system
namespace. In the case PX_SECRETS_NAMESPACE
is set, create the secret in the defined namespace.
Refer to the following example Kubernetes secret specification to create your own secret:
apiVersion: v1
kind: Secret
metadata:
name: px-vault
namespace: portworx
type: Opaque
data:
VAULT_ADDR: <base64 encoded value of the vault endpoint address>
VAULT_BACKEND_PATH: <base64 encoded value of the custom backend path if different than the default "secret">
VAULT_CACERT: <base64 encoded file path where the CA Certificate is present on all the nodes>
VAULT_CAPATH: <base64 encoded file path where the Certificate Authority is present on all the nodes>
VAULT_CLIENT_CERT: <base64 encoded file path where the Client Certificate is present on all the nodes>
VAULT_CLIENT_KEY: <base64 encoded file path where the Client Key is present on all the nodes>
VAULT_TLS_SERVER_NAME: <base64 encoded value of the TLS server name>
VAULT_AUTH_METHOD: YXBwcm9sZQ== # base64 encoded value of "approle"
VAULT_APPROLE_ROLE_ID: <base64 encoded value of the Role ID>
VAULT_APPROLE_SECRET_ID: <base64 encoded value of the Secret ID>
VAULT_NAMESPACE: <base64 encoded value of the global vault namespace for portworx>
For AppRole authentication, the three additioanl parameters are VAULT_AUTH_METHOD
, VAULT_APPROLE_ROLE_ID
, and VAULT_APPROLE_SECRET_ID
.
Step 2: Setup Vault as the secrets provider for Portworx
New Installation
When generating the Portworx Kubernetes specification file, select Vault
from the "Secrets type" list.
Existing Installation
Edit your StorageCluster
object, setting the value of the specs.secretsProvider
field to vault
.
Vault security policies
If you configured Vault strictly with policies, then the Vault token provided to Portworx should follow one of the following policies:
# Read and List capabilities on mount to determine which version of kv backend is supported
path "sys/mounts/"
{
capabilities = ["read", "list"]
}
# V1 backends (Using default backend)
# Provide full access to the portworx subkey
path "secret/*"
{
capabilities = ["create", "read", "update", "delete", "list"]
}
# V1 backends (Using custom backend)
# Provide full access to the portworx subkey
# Provide -> VAULT_BACKEND_PATH=custom-backend (required)
path "custom-backend/*"
{
capabilities = ["create", "read", "update", "delete", "list"]
}
# V2 backends (Using default backend )
# Provide full access to the data/portworx subkey
path "secret/data/*"
{
capabilities = ["create", "read", "update", "delete", "list"]
}
# V2 backends (Using custom backend )
# Provide full access to the data/portworx subkey
# Provide -> VAULT_BACKEND_PATH=custom-backend (required)
path "custom-backend/data/*"
{
capabilities = ["create", "read", "update", "delete", "list"]
}
Portworx supports only the kv backend of Vault.
You can set all the above Vault related fields and the cluster secret key using the Portworx CLI, which is explained in the next section.
Run the following command to create a policy that by using the above policies
vault policy write portworx portworx.hcl
Key generation with Vault
The following sections describe the key generation process with Portworx and Vault, which you can use for encrypting volumes. For more information about encrypted volumes, refer to the Encrypted volumes using pxctl page.
Setting cluster wide secret key
A cluster-wide secret key is a common key that you can use to encrypt all your volumes. Run the following command to set the cluster secret key:
pxctl secrets set-cluster-key --secret <cluster-wide-secret-key>
You should run this command only once for the cluster. If you added the cluster secret key through the config.json
, then the command above overwrites it. Even on subsequent Portworx restarts, the cluster secret key in config.json
will be ignored for the one set through the CLI.
Vault credentials reference
Portworx requires the following Vault credentials to use its APIs:
-
Vault Address [VAULT_ADDR]
Vault server address expressed as a URL and port. For example:
https://192.168.11.11:8200
-
Vault Token [VAULT_TOKEN]
Vault authentication token. Refer to the Vault tokens page for more information about Vault tokens. If you are using the Kubernetes authentication method of Vault, then you need not provide the actual token to Portworx.
-
Vault Backend Path [VAULT_BACKEND_PATH]
The custom backend path if different than the default
secret
-
Vault CA Certificate [VAULT_CACERT]
Path to a PEM-encoded CA certificate file that needs to be present on all Portworx nodes. This file is used to verify the SSL certificate of Vault server. This variable takes precedence over
VAULT_CAPATH
. -
Vault CA Path [VAULT_CAPATH]
Path to a directory of PEM-encoded CA certificate files that needs to be present on all Portworx nodes.
-
Vault Client Certificate [VAULT_CLIENT_CERT]
Path to a PEM-encoded client certificate that needs to be present on all Portworx nodes. This file is used for TLS communication with the Vault server.
-
Vault Client Key [VAULT_CLIENT_KEY]
Path to an unencrypted, PEM-encoded private key which corresponds to the matching client certificate. This key file needs to be present on all Portworx nodes.
-
Vault TLS Server Name [VAULT_TLS_SERVER_NAME]
Name to use as the SNI host when you connect using TLS.
-
Vault Auth Method [VAULT_AUTH_METHOD]
Specifies the authentication method that Portworx should use while authenticating with Vault. "Kubernetes" is the currently supported authenication method.
-
Vault Auth Kubernetes Role [VAULT_AUTH_KUBERNETES_ROLE]
Name of the Kubernetes "Auth Role" created in Vault for Portworx. This field is set only when using the Kubernetes authentication method.
-
Vault Namespace [VAULT_NAMESPACE]
Allows you to set a global Vault namespace for the Portworx cluster. All Vault requests by Portworx use this Vault namespace, if you do not provide an override.
Using Vault with Portworx
📄️ Encrypt Kubernetes PVCs with Vault
Instructions on using Vault with Portworx for encrypting PVCs in Kubernetes