Configure multi-SAN protocol support

Multi-SAN protocol support lets you assign different SAN protocols—iSCSI, NVMe-TCP, NVMe-RoCE, NVMe-FC, or FC—to different nodes in a single Kubernetes cluster. Use this feature when your cluster includes nodes with different hardware capabilities, such as nodes that use iSCSI alongside newer nodes that support NVMe-TCP, or when you want to evaluate or transition between protocols without rebuilding the cluster.

Each node uses only one protocol. The protocol is assigned at node startup based on per-node configuration in the StorageCluster resource. Volume attachment protocol is determined per node at attach time, so existing PVCs work transparently when pods are rescheduled across nodes with different protocols.

Prerequisites

Before you configure multi-SAN protocol support, make sure:

• PX-CSI version is 26.3.0 or later.
• The FlashArray has ports configured and enabled for each protocol you plan to use.
• Each node is prepared for its assigned protocol. See Prepare FlashArray hosts for host-side configuration requirements for each protocol.

How multi-SAN protocol support works

The CSI node plugin DaemonSet reads StorageCluster.spec.nodes[] at startup. Each entry in that list specifies a node selector and a set of environment variable overrides. When the node plugin starts on a node, it matches the node against each selector and applies the first matching entry's environment variable overrides before initializing the transport layer.

For multi-SAN protocol support, you set PURE_FLASHARRAY_SAN_TYPE in spec.nodes[].env to assign a specific protocol to nodes matching each selector. A node-level value for PURE_FLASHARRAY_SAN_TYPE takes precedence over the cluster-level value in spec.env[]. Nodes that do not match any selector in spec.nodes[] use the cluster-level PURE_FLASHARRAY_SAN_TYPE value.

After initializing, the node plugin writes the node's protocol and initiator credentials (IQN for iSCSI, NQN for NVMe, or WWN for FC) to the node's StorageNodeInitiator object. When a volume is attached to a node, the controller reads that node's StorageNodeInitiator to determine which protocol and credentials to use for the attachment request.

Configure multi-SAN protocol support

Use this procedure for a fresh Portworx install, before the CSI node plugin has started on the cluster's nodes. To change the protocol on a node that's already running, see Change the protocol on an existing node instead.

Each entry in spec.nodes[] uses a selector to identify which nodes it applies to. A selector matches nodes either by label, using labelSelector, or by name, using nodeName—use whichever fits how you identify the node:

• labelSelector matches any node carrying the specified label, so it's useful for assigning a protocol to a group of nodes at once.
• nodeName matches a single named node, so it's useful for targeting one node individually without labeling it.

1. (Optional, only needed if you're using labelSelector) Label each node with its target protocol. The label key and value are user-defined—choose values that are meaningful in your environment:

   kubectl label node <node-name> storage-protocol=nvme-tcp

2. Edit the StorageCluster resource to add per-node protocol overrides:

   kubectl edit storagecluster -n portworx

3. Add spec.nodes[] entries, each with a selector and the corresponding PURE_FLASHARRAY_SAN_TYPE override. The example below shows both selector types—a labelSelector matching all nodes labeled storage-protocol: nvme-tcp, and a nodeName targeting one specific node:

   apiVersion: core.libopenstorage.org/v1
   kind: StorageCluster
   metadata:
     name: px-cluster
     namespace: portworx
   spec:
     env:
     - name: PURE_FLASHARRAY_SAN_TYPE
       value: ISCSI  # default for nodes not matched by spec.nodes[]
     nodes:
     - selector:
         labelSelector:
           matchLabels:
             storage-protocol: nvme-tcp
       env:
       - name: PURE_FLASHARRAY_SAN_TYPE
         value: NVMEOF-TCP
     - selector:
         nodeName: node1.example.com
       env:
       - name: PURE_FLASHARRAY_SAN_TYPE
         value: NVMEOF-RDMA

   Valid values for PURE_FLASHARRAY_SAN_TYPE are: ISCSI, FC, NVMEOF-TCP, NVMEOF-RDMA, NVMEOF-FC.

4. Save the changes. When the CSI node plugin pods start on each node, they read the updated StorageCluster configuration and initialize their transport using the assigned protocol.

Change the protocol on an existing node

To change the SAN protocol on a node with running workloads, first move those workloads off the node to avoid disrupting active volume attachments.

note

• A FlashArray host registered with NVMe credentials cannot connect to a volume that was previously attached via iSCSI, and vice versa. Draining the node in step 2 ensures all volume connections are detached before the host's protocol changes.
• Host groups and hosts that are not connected through NVMe cannot connect to volumes that are already connected to NVMe-connected hosts.

1. Cordon the node to prevent new pods from being scheduled on it:

   kubectl cordon <node-name>

2. Drain the node to evict running pods. Use --ignore-daemonsets because DaemonSet pods such as the CSI node plugin cannot be evicted:

   kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

3. Update the StorageCluster resource to set the new protocol for this node:

   spec:
     nodes:
     - selector:
         nodeName: <node-name>
       env:
       - name: PURE_FLASHARRAY_SAN_TYPE
         value: NVMEOF-TCP

4. Save the changes.

5. Restart the CSI node plugin pod on this node manually. The pod does not restart automatically when spec.nodes[].env changes:

   kubectl delete pod -n portworx -l app.kubernetes.io/component=node-plugin --field-selector spec.nodeName=<node-name>

6. Verify that the StorageNodeInitiator for this node reflects the new protocol before uncordoning:

   kubectl get storagenodeinitiator <node-name> -o jsonpath='{.spec.sanTypes}'

7. Uncordon the node to allow pods to be scheduled on it again:

   kubectl uncordon <node-name>

Verify the per-node SAN protocol configuration

After the node plugin pods restart, confirm that each node's StorageNodeInitiator reflects the correct protocol:

kubectl get storagenodeinitiator <node-name> -o jsonpath='{.spec.sanTypes}'

The command returns the protocol assigned to that node, for example:

["NVMEOF-TCP"]

To view all StorageNodeInitiator objects in the cluster and compare protocols across nodes:

kubectl get storagenodeinitiator -o custom-columns='NODE:.metadata.name,PROTOCOL:.spec.sanTypes'

Related topics

• Prepare FlashArray hosts
• StorageCluster CRD reference
• StorageNodeInitiator CRD reference
• Configure subnet-aware iSCSI login