Portworx

Create and manage volumes using pxctl

In this document, we are going to show you how to create and manage volumes with the pxctl command-line tool. Note that you can use the new volumes directly in Docker with the -v option.

To view a list of the available commands, run the following command:

/opt/pwx/bin/pxctl volume --help
Manage volumes

Usage:
  pxctl volume [flags]
  pxctl volume [command]

Aliases:
  volume, v

Examples:
pxctl volume create -s 100 myVolName

Available Commands:
  access               Manage volume access by users or groups
  check                Perform Background filesystem consistency check operation on the volume
  clone                Create a clone volume
  create               Create a volume
  delete               Delete a volume
  ha-update            Update volume HA level
  import               Import data into a volume
  inspect              Inspect a volume
  list                 List volumes in the cluster
  locate               Locate volume
  requests             Show all pending requests
  restore              Restore volume from snapshot
  snap-interval-update Update volume configuration
  snapshot             Manage volume snapshots
  stats                Volume Statistics
  trim                 Frees unused blocks in the volume to the portworx pool
  update               Update volume settings
  usage                Show volume usage information

Flags:
  -h, --help   help for volume

Global Flags:
      --ca string            path to root certificate for ssl usage
      --cert string          path to client certificate for ssl usage
      --color                output with color coding
      --config string        config file (default is $HOME/.pxctl.yaml)
      --context string       context name that overrides the current auth context
  -j, --json                 output in json
      --key string           path to client key for ssl usage
      --output-type string   use "wide" to show more details
      --raw                  raw CLI output for instrumentation
      --ssl                  ssl enabled for portworx

Use "pxctl volume [command] --help" for more information about a command.

In the next sections, we will take a look at these commands individually.

Create volumes

Portworx creates volumes from the global capacity of a cluster. You can expand the capacity and throughput of your cluster by adding new nodes to the cluster. Portworx protects your volumes from hardware and node failures through automatic replication.

Here are a few things you should consider when creating a new volume with the pxctl command-line tool:

  • Durability: Set the replication level through policy, using the High Availability setting. See the storage policy page for more details.
  • Portworx synchronously replicates each write operation to a quorum set of nodes.
  • Portworx uses an elastic architecture. This allows you to add capacity and throughput at every layer, at any time.
  • Volumes are thinly provisioned. They only use as much storage as needed by the container.
  • You can expand and contract the maximum size of a volume, even after you write data to the volume.

You can create a volume before being used by its container. Also, the container can create the volume at runtime. When you create a volume, the pxctl command-line tool returns the ID of the volume. You can see the same volume if ID if you run a Docker command such as docker volume ls.

To create a volume, run the pxctl volume create and pass it the name of the volume. The following example creates a volume called myVol:

pxctl volume create myVol
3903386035533561360
Portworx controls the throughput at per-container level and can be shared. Volumes have fine-grained control, set through policy.

Before you move on, take a bit of time to make sure you understand the following points:

  • Throughput is set by the IO Priority setting. Throughput capacity is pooled.
  • If you add a node to the cluster, you expand the available throughput for read and write operations.
  • Portworx selects the best node to service read operations, no matter that the operation is from local storage devices or the storage devices attached to another node.
  • Read throughput is aggregated, where multiple nodes can service one read request in parallel streams.
  • Fine-grained controls: Policies are specified per volume and give full control to storage.
  • Policies enforce how the volume is replicated across the cluster, IOPs priority, filesystem, blocksize, and additional parameters described below.
  • Policies are specified at create time and can be applied to existing volumes.

The pxctl command-line utility provides a multitude of options for setting the policies on a volume. Let’s get a feel for the available options by running the pxctl volume create with the --help flag:

pxctl volume create --help
Create a volume

Usage:
  pxctl volume create [flags]

Aliases:
  create, c

Examples:
pxctl volume create [flags] volume-name

Flags:
  -a, --aggregation_level string            aggregation level (Valid Values: [1 2 3 auto]) (default "1")
      --async_io                            Enable async IO to backing storage
      --best_effort_location_provisioning   requested nodes, zones, racks are optional
  -b, --block_size uint                     block size in Bytes (default 4096)
      --cow_ondemand                        Enable On-demand COW on volume (default true)
  -d, --daily hh:mm,k                       daily snapshot at specified hh:mm,k (keeps 7 by default)
      --direct_io                           Enable Direct IO on volume
      --early_ack                           Reply to async write requests after it is copied to shared memory
      --enforce_cg                          enforce group during provision
  -g, --group string                        group
  -h, --help                                help for create
      --io_priority string                  IO Priority (Valid Values: [high medium low]) (default "low")
      --io_profile string                   IO Profile (Valid Values: [sequential cms sync_shared db db_remote auto]) (default "auto")
      --journal                             Journal data for this volume
  -l, --label pairs                         list of comma-separated name=value pairs
  -m, --monthly day@hh:mm,k                 monthly snapshot at specified day@hh:mm,k (keeps 12 by default)
      --mount_options string                comma separated list of mount options provided as key=value pairs
      --nodes string                        comma-separated Node Ids or Pool Ids
      --nodiscard                           Disable discard support for this volume
  -p, --periodic mins,k                     periodic snapshot interval in mins,k (keeps 5 by default), 0 disables all schedule snapshots
      --policy string                       policy names separated by comma
      --proxy_endpoint string               proxy endpoint provided in the following format '<protocol>://<endpoint>' Ex 'nfs://<nfs-server-ip>'
      --proxy_nfs_exportpath string         export path for nfs proxy volume
      --proxy_nfs_subpath string            sub path from the nfs share to which this proxy volume has access to
      --proxy_write                         Enable proxy write replication for this volume
  -q, --queue_depth uint                    block device queue depth (Valid Range: [1 256]) (default 128)
      --racks string                        comma-separated Rack names
  -r, --repl uint                           replication factor (Valid Range: [1 3]) (default 1)
      --scale uint                          auto scale to max number (Valid Range: [1 1024]) (default 1)
      --scan_policy string                  Specify filesystem scan policy (Valid Values: [none scan_on_mount repair_on_mount scan_on_next_mount repair_on_next_mount]) (default "none")
      --secret_key string                   secret_key to use to fetch secret_data for the PBKDF2 function
      --secret_options string               Secret options is used to pass specific secret parameters. Usage: --secret_options=k1=v1,k2=v2
      --secure                              encrypt this volume using AES-256
      --shared                              make this a globally shared namespace volume
      --sharedv4                            export this volume via Sharedv4 at /var/lib/osd/exports
      --sharedv4_mount_options string       comma separated list of sharedv4 client mount options provided as key=value pairs
  -s, --size uint                           volume size in GB (default 1)
      --sticky                              sticky volumes cannot be deleted until the flag is disabled
      --storagepolicy string                storage policy name
      --use_cluster_secret                  Use cluster wide secret key to fetch secret_data
  -w, --weekly weekday@hh:mm,k              weekly snapshot at specified weekday@hh:mm,k (keeps 5 by default)
      --zones string                        comma-separated Zone names

Global Flags:
      --ca string            path to root certificate for ssl usage
      --cert string          path to client certificate for ssl usage
      --color                output with color coding
      --config string        config file (default is $HOME/.pxctl.yaml)
      --context string       context name that overrides the current auth context
  -j, --json                 output in json
      --key string           path to client key for ssl usage
      --output-type string   use "wide" to show more details
      --raw                  raw CLI output for instrumentation
      --ssl                  ssl enabled for portworx
These options can also be passed in through the scheduler or using the inline volume spec. See the inline volume spec section below for more details.

Place a replica on a specific volume

Use the --nodes=LocalNode flag to create a volume and place at least one replica of the volume on the node where the command is run. This is useful when you use a script to create a volume locally on a node.

NOTE: You can provide a node ID, node IP address, or pool UUID to the --nodes flag.

As an example, here’s how you can create a volume named localVolume and place a replica of the volume on the local node:

pxctl volume create --nodes=LocalNode localVolume
Volume successfully created: 756818650657204847

Now, you can check that the replica of the volume is on the node where the command was run:

pxctl volume inspect localVolume
Volume  :  756818650657204847
        Name                     :  localVolume
        Size                     :  1.0 GiB
        Format                   :  ext4
        HA                       :  1
        IO Priority              :  LOW
        Creation time            :  Mar 20 00:30:05 UTC 2019
        Shared                   :  no
        Status                   :  up
        State                    :  detached
        Reads                    :  0
        Reads MS                 :  0
        Bytes Read               :  0
        Writes                   :  0
        Writes MS                :  0
        Bytes Written            :  0
        IOs in progress          :  0
        Bytes used               :  340 KiB
        Replica sets on nodes:
                Set 0
                  Node           : 70.0.29.90 (Pool 1)
        Replication Status       :  Detached
The replicas are visible in the Replica sets on nodes section.

Create volumes with Docker

All docker volume commands are reflected in Portworx. For example, a docker volume create command provisions a storage volume in a Portworx storage cluster.

Use the following command to create a volume named testVol:

docker volume create -d pxd --name testVol
testVol

To make sure the command is reflected into Portworx, run:

pxctl volume list --name testVol
ID			NAME		SIZE	HA	SHARED	ENCRYPTED	IO_PRIORITY	STATUS		SNAP-ENABLED
426544812542612832	testVol	1 GiB	1	no	no		LOW		up - detached	no

Add optional parameters with the –opt flag

As part of the docker volume command, you can add optional parameters through the --opt flag. The parameters are the same, whether you use Portworx storage through the Docker volume or the pxctl command.

The following command uses the --opt flag to specify the filesystem of the container and the size of the volume:

docker volume create -d pxd --name opt_example --opt fs=ext4 --opt size=1G
opt_example

Now, let’s check the setting of our newly created volume. Run the pxctl volume list command and pass it the --name flag with the name of the volume:

pxctl volume list --name opt_example
ID			NAME		SIZE	HA	SHARED	ENCRYPTED	IO_PRIORITY	STATUS		SNAP-ENABLED
282820401509248281	opt_example	1 GiB	1	no	no		LOW		up - detached	no

Inline volume spec

With Portworx, you can pass the volume spec inline along with the volume name. This is useful if you want to create a volume with your scheduler application template inline instead of creating it beforehand.

For example, the following command creates volume called demovolume with:

  • IO priority level = high
  • initial size = 10G
  • replication factor = 3
  • periodic and daily snapshots
docker volume create -d pxd io_priority=high,size=10G,repl=3,snap_schedule="periodic=60#4;daily=12:00#3",name=demovolume

You can make it so that Docker starts a specific container dynamically. Use the following command to create a volume dynamically and start the busybox container:

docker run --volume-driver pxd -it -v io_priority=high,size=10G,repl=3,snap_schedule="periodic=60#4;daily=12:00#3",name=demovolume:/data busybox sh
The spec keys must be comma separated.

The pxctl command-line utility provides support for the following key-value pairs:

IO priority      - io_priority=[high|medium|low]
Volume size      - size=[1..9][G|M|T]
HA factor        - repl=[1,2,3]
Block size       - bs=[4096...]
Shared volume    - shared=true
File System      - fs=[xfs|ext4]
Encryption       - passphrase=secret
snap_schedule    - "periodic=mins#k;daily=hh:mm#k;weekly=weekday@hh:mm#k;monthly=day@hh:mm#k" where k is the number of snapshots to retain.

The inline specs can be passed in through the scheduler application template. For example, below is a snippet from a marathon configuration file:

"parameters": [
	{
		"key": "volume-driver",
		"value": "pxd"
	},
	{
		"key": "volume",
		"value": "size=100G,repl=3,io_priority=high,name=mysql_vol:/var/lib/mysql"
	}],

The global namespace

Through shared and sharedv4 volumes (also known as a global namespace), a single volume’s filesystem is concurrently available to multiple containers running on multiple hosts.

Note 1: You do not need to use shared/sharedv4 volumes to have your data accessible on any host in the cluster. Any Portworx volumes can be exclusively accessed from any host as long as they are not simultaneously accessed. Shared volumes are for providing simultaneous (concurrent or shared) access to a volume from multiple hosts at the same time.
Note 2: You do not necessarily need a replication factor of greater than 1 on your volume in order for it to be shared. Even a volume with a replication factor of 1 can be shared on as many nodes as there are in your cluster.

A typical pattern is for a single container to have one or more volumes. Conversely, many scenarios would benefit from multiple containers being able to access the same volume, possibly from different hosts. Accordingly, the shared volume feature enables a single volume to be read/write accessible by multiple containers. Example use cases include:

  • A technical computing workload sourcing its input and writing its output to a shared volume.
  • Scaling a number of Wordpress containers based on load while managing a single shared volume.
  • Collecting logs to a central location
Note: Usage of shared/sharedv4 volumes for databases is not recommended, since they have a small metadata overhead. Along with that, typical databases do not support concurrent writes to the underlying database at the same time.

Difference between shared and sharedv4 volumes, is the underlying protocol that is used to share this global namespace across multiple hosts.

Delete volumes

You can delete a volume by running the pxctl volume delete with the name of the volume you want to delete:

pxctl volume delete myOldVol
Delete volume 'myOldVol', proceed ? (Y/N): y
Volume myOldVol successfully deleted.

Import volumes

You can import files from a directory into an existing volume. Files already existing on the volume will be retained or overwritten.

As an example, to import the files from /path/to/files into myVol, run the pxctl volume import and pass it the --src flag as in the following example:

pxctl volume import --src /path/to/files myVol
Starting import of  data from /path/to/files into volume myVol...Beginning data transfer from /path/to/files myVol
Imported Bytes :   0% [>---------------------------------------------------------------------------------------------------------------------------------------] 14ms
Imported Files :   0% [>---------------------------------------------------------------------------------------------------------------------------------------] 16ms

Volume imported successfully

Inspect volumes

Click on the section below for instructions on how to inspect your Portworx volumes.

link Inspect volumes

List volumes

To list all volumes within a cluster, use this command:

pxctl volume list
ID			NAME		SIZE	HA	SHARED	ENCRYPTED	IO_PRIORITY	STATUS				SNAP-ENABLED
951679824907051932	objectstorevol	10 GiB	1	no	no		LOW		up - attached on 192.168.99.101	no
810987143668394709	testvol		1 GiB	1	no	no		LOW		up - detached			no
1047941676033657203	testvol2	1 GiB	1	no	no		LOW		up - detached			no
800735594334174869	testvol3	1 GiB	1	no	no		LOW		up - detached			no

Locate volumes

The pxctl volume locate command shows where a given volume is mounted in the containers running on the node:

pxctl volume locate 794896567744466024
host mounted:
  /directory1
  /directory2

In this example, the volume is mounted in two containers via the /directory1 and /directory2 mount points.

Create volume snapshots

Snapshots are efficient point-in-time read-only copies of volumes. Once created, you can use a snapshot to read data, restore data, and to make clones from a given snapshot.

Under the hood, snapshots are using a copy-on-write technique, so that they store only the modified data. This way, snapshots significantly reduce the consumption of resources.

Snapshots can be created explicitly by running the pxctl volume snapshot create command (called henceforth user created snapshots) or through a schedule that is set on the volume.

Here’s an example of how to create a snapshot:

pxctl volume snapshot create --name mysnap --label color=blue,fabric=wool myvol
Volume snap successful: 234835613696329810

The string of digits in the output is the volume ID of the new snapshot. You can use this ID(234835613696329810) or the name(mysnap), to refer to the snapshot in subsequent pxctl commands.

The label values allow you to tag the snapshot with descriptive information of your choosing. You can use them to filter the output of the pxctl volume list command.

There is an implementation limit of 64 snapshots per volume.

Snapshots are read-only. To restore a volume from a snapshot, use the pxctl volume restore command.

  • For information about creating snapshots of your Portworx volumes through Kubernetes, refer to the Create and use snapshots page.

Clone volumes

Use the pxctl volume clone command to create a volume clone from a volume or snapshot. You can refer to the in-built help, by running the pxctl volume clone command with the --help flag:

pxctl volume clone --help
Create a clone volume

Usage:
  pxctl volume clone [flags]

Aliases:
  clone, cl

Examples:
pxctl volume clone [flags] volName

Flags:
  -h, --help           help for clone
  -l, --label string   list of comma-separated name=value pairs
      --name string    clone name

Global Flags:
      --ca string            path to root certificate for ssl usage
      --cert string          path to client certificate for ssl usage
      --color                output with color coding
      --config string        config file (default is $HOME/.pxctl.yaml)
      --context string       context name that overrides the current auth context
  -j, --json                 output in json
      --key string           path to client key for ssl usage
      --output-type string   use "wide" to show more details
      --raw                  raw CLI output for instrumentation
      --ssl                  ssl enabled for portworx

As an example, here’s how you can make a clone named myvol_clone from the parent volume `myvol:

pxctl volume clone -name myvol_clone myvol
Volume clone successful: 55898055774694370
  • For information about creating a clone from a snapshot through Kubernetes, refer to the On-demand snapshots page.

Restore a volume

In order to restore a volume from snapshot use the pxctl volume restore command:

/opt/pwx/bin/pxctl volume restore -h
Restore volume from snapshot

Usage:
  pxctl volume restore [flags]

Aliases:
  restore, r

Examples:
pxctl volume restore [flags] volName

Flags:
  -h, --help              help for restore
  -s, --snapshot string   snapshot-name-or-ID

Global Flags:
      --ca string            path to root certificate for ssl usage
      --cert string          path to client certificate for ssl usage
      --color                output with color coding
      --config string        config file (default is $HOME/.pxctl.yaml)
      --context string       context name that overrides the current auth context
  -j, --json                 output in json
      --key string           path to client key for ssl usage
      --output-type string   use "wide" to show more details
      --raw                  raw CLI output for instrumentation
      --ssl                  ssl enabled for portworx

In the below example parent volume myvol is restored from its snapshot mysnap. Make sure volume is detached in order to restore from the snapshot.

pxctl volume restore --snapshot mysnap myvol
Successfully started restoring volume myvol from mysnap.
  • For information about restoring a Portworx volume with data from a snapshot through Kubernetes, refer to the Restore snapshots page.

Update the snap interval of a volume

Please see the documentation for snapshots for more details.

  • For information about creating scheduled snapshots of a Portworx volume through Kubernetes, refer to the Scheduled snapshots page.

Show volume stats

The pxctl volume stat command shows the real-time read/write IO throughput:

pxctl volume stats mvVol
TS			Bytes Read	Num Reads	Bytes Written	Num Writes	IOPS		IODepth		Read Tput	Write Tput	Read Lat(usec)	Write Lat(usec)
2019-3-4:11 Hrs		0 B		0		0 B		0		0		0		0 B/s		0 B/s		0		0

Manage volume access rules

With pxctl, you can manage your volume access rules. See the volume access page for more details.

Update the replication factor of a volume

You can use the pxctl volume ha-update to increase or decrease the replication factor of a Portworx volume. Consult the update volumes page for more details.

Volume pending requests

Run the following command to show all pending requests:

pxctl volume requests
Only support getting requests for all volumes.
Active requests for all volumes: count = 0

Update the settings of a volume

With the pxctl volume update command, you can update the settings of your Portworx volumes. Consult the updating volumes page for additional details.

Volume usage

To get extended info about the usage of your Portworx volumes, run the pxctl volume usage command with the name or the ID of your volume:

pxctl volume usage 13417687767517527

Understand copy-on-write features

By default, Portworx uses features present in the underlying file system to take snapshots through copy-on-write and checksum storage blocks.

When using the copy-on-write feature to take snapshots, overwriting a block does not update it in place. Instead, every overwrite allocates or updates a new block, and the filesystem metadata is updated to point to this new block. This technique is called redirect-on-write. When using this feature, a block overwrite almost always involves block updates in multiple areas: the target block, any linked indirect file blocks, filesystem metadata blocks, and filesystem metadata indirect file blocks. In a background process separate from the overwrite operation, the old block is freed only if it’s not being referenced by a snapshot/clone.

Alongside copy-on-write, the file system checksums all blocks to detect lost writes and stores the checksum values in a different location away from their associated data.

While these combined features increase the integrity of the data stored on the filesystem, they also increase the read and write overhead on the drives that use them, slowing down performance and increasing latency during file operations.

Depending on your use-case, you may wish to trade off the integrity copy-on-write features offer for increased performance and lower latency. You can do so on a per-volume basis through the pxctl command.

Disable copy-on-write features for a volume

Enter the pxctl volume update command with the --cow_ondemand off flag, followed by the ID of the volume you want to disable copy-on-write features for:

  pxctl volume update  --cow_ondemand off <volume-ID>
  Update Volume: Volume update successful for volume 850767800314736346
Last edited: Friday, Mar 19, 2021
Questions? Visit the Portworx forum.