Skip to main content
Version: 3.6

Installation on an AWS Gardener Cluster

This topic provides instructions for installing Portworx Enterprise on a Gardener cluster running on AWS. Ensure that your cluster meets all the prerequisites before installing Portworx Enterprise.

The following collection of tasks describes how to install Portworx Enterprise on an AWS Gardener cluster:

Complete all the tasks to install Portworx.

Create a Shoot Cluster​

Use the Gardener dashboard or the Gardener API to create a shoot cluster on AWS. The shoot specification controls the worker node EC2 instances, networking, and the OIDC issuer that Portworx will federate against when using Workload Identity.

  1. Sign in to your Gardener dashboard, or set up kubectl access to your Gardener landscape.

  2. Create a shoot cluster on the aws provider. In the shoot specification, ensure the following fields are set:

    • spec.cloudProfile.name: aws
    • spec.region — set to the AWS region where Portworx will run.
    • spec.provider.workers — define the worker pools. Each worker pool's EC2 instances are launched through Auto Scaling groups that Portworx will be granted access to.

  3. Once the shoot is reconciled, download its kubeconfig from the Gardener dashboard or via kubectl, and confirm you can reach the cluster:

    kubectl get nodes

  4. Note the following values from your shoot. These values are required when you configure the authentication:

    • The AWS account ID hosting the shoot.
    • The IAM role (instance profile) attached to the shoot's worker node EC2 instances.
    • The OIDC issuer URL of the shoot (only required for Workload Identity).

For more information on shoot creation, see the Gardener documentation.

Configure Authentication​

Portworx supports two authentication methods on Gardener:

  • IAM Policy - attach an IAM policy with the required permissions to the Gardener worker nodes' instance role. Portworx uses these node-level credentials at runtime.
  • Workload Identity (IRSA) - federated identity using AWS IAM Roles for Service Accounts (IRSA). The eks-pod-identity-webhook injects a projected service account token and the environment variables required by the AWS SDK into Portworx pods, so no static credentials are stored in the cluster. For more information, see Workload identity for cloud operations in Portworx.
LIMITATION

Portworx Enterprise does not support AWS KMS grants.

Choose the method that matches your security and operational requirements.

With IAM Policy authentication, Portworx uses the credentials of the IAM role attached to the Gardener worker node EC2 instances. Create an IAM policy with the permissions Portworx requires and attach it to the worker node instance role.

  1. In the AWS Management Console, go to IAM > Policies and select Create policy.

  2. Choose the JSON tab, and then paste the following permissions into the editor:

    note

    These are the minimum permissions required for storage operations for a Portworx cluster. For the complete set of permissions for all Portworx storage operations, see the credentials reference.

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "EBSVolumeManagement",
                "Effect": "Allow",
                "Action": [
                    "ec2:CreateVolume",
                    "ec2:DeleteVolume",
                    "ec2:AttachVolume",
                    "ec2:DetachVolume",
                    "ec2:DescribeVolumes",
                    "ec2:ModifyVolume",
                    "ec2:DescribeVolumeStatus",
                    "ec2:CreateSnapshot",
                    "ec2:DeleteSnapshot",
                    "ec2:DescribeSnapshots",
                    "ec2:CreateTags",
                    "ec2:DeleteTags",
                    "ec2:DescribeTags",
                    "ec2:DescribeVolumeAttribute",
                    "ec2:DescribeVolumesModifications",
                    "autoscaling:DescribeAutoScalingGroups"
                ],
                "Resource": "*"
            },
            {
                "Sid": "InstanceDescription",
                "Effect": "Allow",
                "Action": [
                    "ec2:DescribeInstances"
                ],
                "Resource": "*"
            }
        ]
    }

  3. Name the policy and create it.

  4. Attach the policy to the IAM role (instance profile) of your Gardener worker nodes:

    1. From the IAM page, select Roles in the left pane.
    2. Search for and select the worker node instance role of your shoot cluster.
    3. From the Add permissions dropdown, select Attach policies.
    4. Search for the policy you created, select it, and then select Attach policies.
    note

    Gardener may recreate worker nodes during shoot reconciliation (for example, after a worker pool update). The IAM policy remains attached because it is bound to the worker node instance role, not to individual instances.

Install Node Packages​

Portworx requires the following operating system packages on every worker node before installation:

  • lvm2
  • mdadm
  • parted
  • augeas-tools
  • thin-provisioning-tools
  • dmsetup
  • nfs-kernel-server
  • nfs-common
  • rpcbind

Portworx does not install operating system packages on Garden Linux worker nodes. If your Gardener cluster uses Garden Linux, ensure that these packages are preinstalled in the worker node image, or provisioned through your supported Gardener or organizational node provisioning process, before you install Portworx. For more information, contact Portworx Support.

Install Portworx​

You can install Portworx Enterprise on an AWS Gardener cluster either by generating a Kubernetes specification through Portworx Central and applying it, or by deploying the Portworx Helm chart. Choose the installation mode that matches your tooling and operational preferences.

To install Portworx, you must first generate Kubernetes manifests that you will deploy in your AWS Gardener cluster by following these steps.

  1. Sign in to the Portworx Central console.
    The system displays the Welcome to Portworx Central! page.

  2. In the Portworx Enterprise section, select Generate Cluster Spec.
    The system displays the Generate Spec page.

  3. From the Portworx Version dropdown menu, select the Portworx version to install.

  4. From the Platform dropdown menu, select AWS.

  5. From the Distribution Name dropdown menu, select Gardener.

  6. In the AWS Role ARN field, enter the ARN of the IAM role you created in the Configure Authentication section, in the format arn:aws:iam::<account-id>:role/<role-name>.
    Portworx Enterprise uses this role to access AWS services through IRSA.

    note

    The default Portworx Central workflow for installing Portworx Enterprise on an AWS Gardener cluster uses workload identity (IRSA). If you configured IAM Policy authentication instead, leave this field blank and proceed to the Customize step (Step 8) to modify the generated StorageCluster specification for IAM Policy authentication.

  7. In the Namespace field, enter the namespace where you plan to install Portworx.
    By default, the namespace is portworx.

  8. (Optional) To customize the configuration options and generate a custom specification, click Customize and perform the following steps:

    note

    To continue without customizing the default configuration or generating a custom specification, proceed to Step 9.

  • Basic tab:
    1. Select one of the following:
    • To use an existing etcd cluster, do the following:
      1. Select the Your etcd details option.
      2. In the field provided, enter the host name or IP and port number. For example, http://test.com.net:1234.
        To add another etcd cluster, click the + icon.
        note

        You can add up to three etcd clusters.

      3. Select one of the following authentication methods:
        • Disable HTTPS – To use HTTP for etcd communication.
        • Certificate Auth – To use HTTPS with an SSL certificate.
          For more information, see Secure your etcd communication.
        • Password Auth – To use HTTPS with username and password authentication.
    • To use an internal Portworx-managed key-value store (kvdb), do the following:
      1. Select the Built-in option.
      2. TLS for internal KVDB is enabled, by default. If Cert-Manager is already running in your Kubernetes cluster, deselect the Deploy Cert-Manager for TLS certificates option to avoid installation failures.
    1. Click Next.
  • Storage tab:
    warning

    Do not add volumes of different types when configuring storage devices. For example, do not add both GP2 and GP3 or IO1. This can cause performance issues or errors.

    1. Select one of the following:
    • To enable Portworx to provision drives using a specification, do the following:
      1. Select the Create Using a Spec option.
      2. (Optional) To designate PX-StoreV1 as the datastore, select PX-StoreV1. By default, the system selects PX-StoreV2 as the datastore.
      3. (Optional) Select the Run on Small Node Configuration checkbox if your cluster nodes have limited resources (for example, 4 CPU cores and 8 GB of memory).
        Portworx Enterprise requires 8 CPU cores and 8 GB of memory by default. Enabling this option allows Portworx to run on smaller nodes, but it may reduce overall performance compared to the default configuration.
        note

        This checkbox is available only when you select PX-StoreV2 as the datastore.

      4. To add one or more cloud storage drive types for Portworx to use, click + Add Drive and select one of the following types of drives:
        • GP2
        • GP3
        • IO1
        note
        • To select GP2 as the drive type, you must select PX-StoreV1 as the datastore.
        • For PX-StoreV2, four drives are recommended for optimal performance.
      5. Configure the following fields for the drive:
        • Size (GB) - Specify the size of the drive in gigabytes.
        • IOPS required from EBS volume - Enter the input/output operations per second (IOPS) value for the drive.
          note
          • IOPS is required when you select IO1 drive type only.
          • If you do not specify an IOPS value for GP3, Portworx uses the default value of 3000.
        • Throughput for EBS volume - Enter the required data transfer rate for the drive.
        • Encryption - Choose None to disable encryption or BYOK Encryption to encrypt your AWS cluster data disk using BYOK encryption.
        • Encryption Key - If you choose BYOK Encryption, specify the key to use for BYOK encryption.
          For more information, see AWS KMS.
        • Drive Tags - Add labels in key:value format to organize and identify drives.
          This is useful for policies and workload mapping.
        • Action - Use the trash icon to remove a drive type from the configuration.
      6. Initial Storage Nodes (Optional): Enter the number of storage nodes that need to be created across zones and node pools.
      7. From the Default IO Profile dropdown menu, select Auto.
        This enables Portworx to automatically choose the best I/O profile based on detected workload patterns.
      8. From the Journal Device dropdown menu, select one of the following:
        • None – To use the default journaling setting.
        • Auto – To automatically allocate journal devices.
        • Custom – To manually enter a journal device path.
          Enter the path of the journal device in the Journal Device Path field.
    • To enable Portworx to use all available, unused, and unmounted drives on the node, do the following:
      1. Select the Consume Unused option.
      2. (Optional) To designate PX-StoreV1 as the datastore, select PX-StoreV1. By default, the system selects PX-StoreV2 as the datastore.
      3. (Optional) Select the Run on Small Node Configuration checkbox if your cluster nodes have limited resources (for example, 4 CPU cores and 8 GB of memory).
        Portworx requires 8 CPU cores and 8 GB of memory by default. Enabling this option allows Portworx to run on smaller nodes, but it may reduce overall performance compared to the default configuration.
        note

        This checkbox is available only when you select PX-StoreV2 as the datastore.

      4. For PX-StoreV2, in the Metadata Path field, enter a pre-provisioned path for storing the Portworx metadata.
        The path must be at least 64 GB in size.
      5. From the Journal Device dropdown menu, select one of the following:
        • None – To use the default journaling setting.
        • Auto – To automatically allocate journal devices.
        • Custom – To manually enter a journal device path.
          Enter the path of the journal device in the Journal Device Path field.
      6. Select the Use unmounted disks even if they have a partition or filesystem on it. Portworx will never use a drive or partition that is mounted checkbox to use unmounted disks, even if they contain a partition or filesystem.
        Portworx will not use any mounted drive or partition.
    • To enable Portworx to use existing drives on a node, do the following:
      1. Select the Use Existing Drives option.
      2. (Optional) To designate PX-StoreV1 as the datastore, select PX-StoreV1. By default, the system selects PX-StoreV2 as the datastore.
      3. (Optional) Select the Run on Small Node Configuration checkbox if your cluster nodes have limited resources (for example, 4 CPU cores and 8 GB of memory).
        Portworx requires 8 CPU cores and 8 GB of memory by default. Enabling this option allows Portworx to run on smaller nodes, but it may reduce overall performance compared to the default configuration.
        note

        This checkbox is available only when you select PX-StoreV2 as the datastore.

      4. For PX-StoreV2, in the Metadata Path field, enter a pre-provisioned path for storing the Portworx metadata.
        The path must be at least 64 GB in size.
      5. In the Drive/Device field, specify the block drive(s) that Portworx uses for data storage.
      6. In the Pool Label field, assign a custom label in key:value format to identify and categorize storage pools.
      7. From the Journal Device dropdown menu, select one of the following:
        • None – To use the default journaling setting.
        • Auto – To automatically allocate journal devices.
        • Custom – To manually enter a journal device path.
          Enter the path of the journal device in the Journal Device Path field.
    1. Click Next.
  • Network tab:
    1. In the Interface(s) section, do the following:
      1. Enter the Data Network Interface to be used for data traffic.
      2. Enter the Management Network Interface to be used for management traffic.
    2. In the Advanced Settings section, do the following:
      1. Enter the Starting port for Portworx services.
        By default, the starting port is 9001.
    3. Select Next.
  • Deployment tab:
    1. In the Kubernetes Distribution section, under Are you running on either of these?, select Gardener.
    2. In the Component Settings section:
      1. (Optional) Select the Enable Stork checkbox to enable Stork.
      2. (Optional) Select the Restrict Data Protection RBAC checkbox to restrict RBAC permissions for Stork (if enabled) and Operator. You will not be able to use Backup and DR capabilities with this restriction. For more information, see Restrict Data Protection RBAC.
      3. (Optional) Select the Enable Monitoring checkbox to enable monitoring of Portworx components and resources.
        • To configure the monitoring stack, select one of the following:
          • Portworx Managed - To enable Portworx to install and manage Prometheus and Operator automatically.
            Ensure that no other Prometheus Operator instance is already running on the cluster.
          • User Managed - To configure and manage your own monitoring stack.
        • Select the Enable Autopilot checkbox to enable Portworx Autopilot. For User Managed monitoring stack, Portworx supports the following metrics providers that Autopilot will use to fetch metrics for rule evaluation and automated actions.
        note

        This checkbox is available only when you select the Enable Monitoring checkbox.

      4. (Optional) Select the Enable Telemetry checkbox to enable telemetry in the StorageCluster spec.
        For information, see Portworx Telemetry.
      5. Enter the prefix for the Portworx cluster name in the Cluster Name Prefix field.
      6. Select the Secrets Store Type from the dropdown menu to store and manage secure information for features such as CloudSnaps and Encryption.
    3. In the Identity and Access Management (IAM) section, select the Enable Workload Identity (WLI) checkbox to use a federated identity with short-lived tokens injected per-pod by the eks-pod-identity-webhook.
      Workload Identity Federation allows the Portworx Operator to configure Portworx components to securely authenticate with cloud services, without storing static cloud credentials. For more information, see Workload identity for cloud operations in Portworx.
      When you enable Enable Workload Identity (WLI):
      • Select the Enable for Stork (for data protection using Portworx DR or Backup) checkbox to enable workload identity for Stork.
        You must select this checkbox to use Portworx Backup or Portworx DR.
      • Under Configure Workload Identity Settings, the Cloud Provider (AWS) and Key (eks.amazonaws.com/role-arn) fields are pre-populated. In the AWS Role ARN field, enter the ARN of the IAM role you created in the Configure Authentication section.
    4. In the Environment Variables section, enter name-value pairs in the respective fields.
    5. In the Registry and Image Settings section:
      1. Enter the Custom Container Registry Location to download the Docker images.
      2. Enter the Kubernetes Docker Registry Secret that serves as the authentication to access the custom container registry.
      3. From the Image Pull Policy dropdown menu, select Default, Always, IfNotPresent, or Never.
        This policy influences how images are managed on the node and when updates are applied.
    6. In the Security Settings section, select the Enable Authorization checkbox to enable Role-Based Access Control (RBAC) and secure access to storage resources in your cluster.
    7. Click Finish.
    8. In the summary page, enter a name for the specification in the Spec Name field, and tags in the Spec Tags field.
    9. Click Download .yaml to download the yaml file with the customized specification or Save Spec to save the specification.
  1. Click Save & Download to generate the specification.

Deploy Portworx Operator​

Use the Operator specification you generated to deploy the Portworx Operator:

kubectl apply -f 'https://install.portworx.com/<version-number>?comp=pxoperator'
serviceaccount/portworx-operator created
podsecuritypolicy.policy/px-operator created
clusterrole.rbac.authorization.k8s.io/portworx-operator created
clusterrolebinding.rbac.authorization.k8s.io/portworx-operator created
deployment.apps/portworx-operator created

Deploy StorageCluster​

Apply the StorageCluster specification you generated:

kubectl apply -f px-storagecluster.yaml
storagecluster.core.libopenstorage.org/px-cluster-xxxxxxxx-xxxx-xxxx-xxxx-8dfd338e915b created

Monitor Portworx Nodes​

  1. Enter the following kubectl get command and wait until all Portworx nodes show as Ready or Online in the output:

    kubectl -n <px-namespace> get storagenodes -l name=portworx
    NAME                   ID                                     STATUS   VERSION   AGE
    gardener-shoot-node0   xxxxxxxx-xxxx-xxxx-xxxx-43cf085e764e   Online   3.6.2     4m52s
    gardener-shoot-node1   xxxxxxxx-xxxx-xxxx-xxxx-4597de6fdd32   Online   3.6.2     4m52s
    gardener-shoot-node2   xxxxxxxx-xxxx-xxxx-xxxx-e2169ffa111c   Online   3.6.2     4m52s

  2. Enter the following kubectl describe command with the NAME of one of the Portworx nodes you retrieved above to show the current installation status for individual nodes:

    kubectl -n <px-namespace> describe storagenode <portworx-node-name>

Verify Portworx Pod Status​

Enter the following command to list the Portworx pods in the namespace where you deployed Portworx:

kubectl get pods -n <px-namespace> -o wide | grep -e portworx -e px
NAME                                                    READY   STATUS    RESTARTS         AGE     IP                NODE                   NOMINATED NODE   READINESS GATES
portworx-api-774c2                                      1/1     Running   0                2m55s   192.168.121.196   username-k8s1-node0    <none>           <none>
portworx-api-t4lf9                                      1/1     Running   0                2m55s   192.168.121.99    username-k8s1-node1    <none>           <none>
portworx-api-dvw64                                      1/1     Running   0                2m55s   192.168.121.99    username-k8s1-node2    <none>           <none>
portworx-kvdb-94bpk                                     1/1     Running   0                4s      192.168.121.196   username-k8s1-node0    <none>           <none>
portworx-kvdb-8b67l                                     1/1     Running   0                10s     192.168.121.196   username-k8s1-node1    <none>           <none>
portworx-kvdb-fj72p                                     1/1     Running   0                30s     192.168.121.196   username-k8s1-node2    <none>           <none>
portworx-operator-58967ddd6d-kmz6c                      1/1     Running   0                4m1s    10.244.1.99       username-k8s1-node0    <none>           <none>
prometheus-px-prometheus-0                              2/2     Running   0                2m41s   10.244.1.105      username-k8s1-node0    <none>           <none>
px-cluster-xxxxxxxx-xxxx-xxxx-xxxx-3e9bf3cd834d-9gs79   2/2     Running   0                2m55s   192.168.121.196   username-k8s1-node0    <none>           <none>
px-cluster-xxxxxxxx-xxxx-xxxx-xxxx-3e9bf3cd834d-vpptx   2/2     Running   0                2m55s   192.168.121.99    username-k8s1-node1    <none>           <none>
px-cluster-xxxxxxxx-xxxx-xxxx-xxxx-3e9bf3cd834d-bxmpn   2/2     Running   0                2m55s   192.168.121.191   username-k8s1-node2    <none>           <none>
px-csi-ext-868fcb9fc6-54bmc                             4/4     Running   0                3m5s    10.244.1.103      username-k8s1-node0    <none>           <none>
px-csi-ext-868fcb9fc6-8tk79                             4/4     Running   0                3m5s    10.244.1.102      username-k8s1-node2    <none>           <none>
px-csi-ext-868fcb9fc6-vbqzk                             4/4     Running   0                3m5s    10.244.3.107      username-k8s1-node1    <none>           <none>
px-prometheus-operator-59b98b5897-9nwfv                 1/1     Running   0                3m3s    10.244.1.104      username-k8s1-node0    <none>           <none>

Note the name of a px-cluster pod. You will run pxctl commands from these pods in Verify Portworx Cluster Status.

If you configured Workload Identity authentication, perform the following additional checks:

  • Verify that Portworx Operator added the IRSA annotation on the portworx service account:

    kubectl -n <px-namespace> get sa portworx -o yaml

    The output should include the following annotation, where the value matches the ARN of the IAM role you created in Configure Authentication:

    metadata:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/<role-name>

  • After the configuration is applied, the Portworx Operator restarts the pods as needed.
    Verify that the Portworx pods include the projected token volume, volume mount, and environment variables injected by the eks-pod-identity-webhook:

    kubectl -n <px-namespace> get pods <px-pod> -oyaml

    Token volume​

    volumes:
    - name: aws-iam-token
      projected:
        defaultMode: 420
        sources:
        - serviceAccountToken:
            audience: sts.amazonaws.com
            expirationSeconds: 86400
            path: token

    Volume mount​

    volumeMounts:
    - mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
      name: aws-iam-token
      readOnly: true

    Environment variables​

    spec:
      containers:
      - env:
        - name: AWS_ROLE_ARN
          value: arn:aws:iam::<account-id>:role/<role-name>
        - name: AWS_WEB_IDENTITY_TOKEN_FILE
          value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
        - name: AWS_DEFAULT_REGION
          value: <region>
        - name: AWS_STS_REGIONAL_ENDPOINTS
          value: regional

    These are used by the AWS SDK to authenticate and make API calls.

Verify Portworx Cluster Status​

You can find the status of the Portworx cluster by running pxctl status from a Portworx pod. Use the pod name you retrieved in Verify Portworx Pod Status:

kubectl exec <pod-name> -n <px-namespace> -- /opt/pwx/bin/pxctl status

Status: PX is operational indicates that the cluster is running as expected. If the cluster is using the PX-StoreV2 datastore, the StorageNode entries for each node display Yes(PX-StoreV2).

Verify Portworx Pool Status​

note

This procedure is applicable for clusters with the PX-StoreV2 datastore.

Run the following command to view the Portworx drive configuration for your pod:

kubectl exec <px-pod> -n <px-namespace> -- /opt/pwx/bin/pxctl service pool show

The output Type: PX-StoreV2 confirms that the pod uses the PX-StoreV2 datastore.

Verify pxctl Cluster Provision Status​

  1. Run the following command to find the storage cluster:

    kubectl -n <px-namespace> get storagecluster

    The status must display the cluster as Online.

  2. Run the following command to find the storage nodes:

    kubectl -n <px-namespace> get storagenodes

    The status must display the nodes as Online.

  3. Verify the Portworx cluster provision status by running the following command. Specify the pod name you retrieved in Verify Portworx Pod Status:

    kubectl exec <px-pod> -n <px-namespace> -- /opt/pwx/bin/pxctl cluster provision-status

What to do next​

Last updated on
In this topic: