Skip to main content

Set up FlashArray NVMe/RoCE

This document explains setup and configuration steps for using Portworx version 2.13.0 or newer on FlashArray NVMe/RoCE (RDMA over Converged Ethernet). Using this feature, you can attach FlashArray volumes using the NVMe/RoCE protocol. Follow the steps in this document when you set up and install Portworx.

NVMe/RoCE can be used with FlashArray Cloud Drives (FACD) or FlashArray Direct Access (FADA) volumes. When you select NVME-oF RMDA during StorageCluster spec generation or specify the protocol in the spec manually, Portworx recognizes that you want to use the NVMe/RoCE protocol and uses it to communicate with the FlashArray.

  • QoS (IOPS and bandwidth) limits are not supported with NVMe volumes.
  • In-place upgrades from iSCSI or Fibre Channel to NVMe are not supported. Changing the SAN type might result in unpredictable attachment behavior.


  • Check that your setup meets the requirements in the NVMe/RoCE Support Matrix.
  • Make sure that your Linux kernel supports NVMe. You need to load the nvme-fabrics and nvme-rdma modules on boot or include them when you compile the kernel.
  • Install the nvme-cli package.
  • Ensure that all nodes have unique NQN (/etc/nvme/hostnqn) and host ID (/etc/nvme/hostid) entries.

Configure hardware

Configure your Cisco, Juniper, or Arista switch for use with Pure FlashArray NVMe/RoCE.

Configure software

To configure your software settings, follow the steps in Linux Recommended Settings, with the following change.

When you set up your multipath.conf file, add a blacklist section as shown in the following example:

blacklist {
devnode "^pxd[0-9]*"
devnode "^pxd*"
device {
vendor "VMware"
product "Virtual disk"

defaults {
polling_interval 10

devices {
device {
vendor "NVME"
product "Pure Storage FlashArray"
path_selector "queue-length 0"
path_grouping_policy group_by_prio
prio ana
failback immediate
fast_io_fail_tmo 10
user_friendly_names no
no_path_retry 0
features 0
dev_loss_tmo 60
find_multipaths yes
device {
vendor "PURE"
product "FlashArray"
path_selector "service-time 0"
hardware_handler "1 alua"
path_grouping_policy group_by_prio
prio alua
failback immediate
path_checker tur
fast_io_fail_tmo 10
user_friendly_names no
no_path_retry 0
features 0
dev_loss_tmo 600
find_multipaths yes

Install Portworx

Perform the steps in one of the following documents to install Portworx version 2.13.0 or newer on your FlashArray setup, then return to this document.

Configure the adapter as a PCI device

Configure the NVMe/RoCE adapter installed in ESXi as a PCI device. For example, on vSphere, follow the steps in Enable Passthrough for a Network Device on a host from the VMware documentation.

Once the NVMe/RoCE adapter is set up as a PCI device, the VM can mount as a PCI device and access external storage directly.

Use NVMe/RoCE in a VM

If you are using a VM, you also need to perform additional steps to enable and configure PCI passthrough.

The following examples illustrate how to perform these steps for vSphere. Your environment might require different steps.

Enable RoCE as PCI passthrough

After you install a physical adapter, the NVMe/RoCE adapter should be listed in PCI Devices.

  1. Navigate to a host in the vSphere Client navigator.

  2. Select the Configure tab, then under Hardware, select PCI devices.

  3. Select all of the NVMe adapters that you have added, then select Toggle passthrough.

    When passthrough configurations complete successfully, the device is listed in the Passthrough-enabled devices tab.

Configure PCI passthrough in a VM

  1. In the vSphere client, select the VM you want to add the PCI passthrough card to from the list of VMs. Right click the VM, then select Edit settings.

  2. Click Add new device, then select PCI device.

  3. Select DirectPathIO, then select any of the RoCE adapter interfaces.

    Add as many PCI devices for RoCE adapters as you need for the VM. Multiple ports on the FlashArray will provide redundant connections, but for extra redundancy you should add two or more PCI devices in case one device fails.

Further considerations

Before upgrading NVMe software, Portworx by Pure Storage recommends putting Portworx in maintenance mode.

Was this page helpful?