Prerequisites

Key-value store

Portworx uses a key-value store for it’s clustering metadata. Please have a clustered key-value database (etcd or consul) installed and ready. For etcd installation instructions please refer to this doc.

Storage

At least one of the Portworx nodes should have extra storage available, in a form of unformatted partition or a disk-drive.

Storage devices explicitly given to Portworx will be automatically formatted by PX.

Shared mounts

If you are running Docker v1.12, you must configure Docker to allow shared mounts propagation (see instructions), as otherwise Portworx will fail to start.

Firewall

Ensure ports 9001-9015 are open between the nodes that will run Portworx.

NTP

Ensure all nodes running PX are time-synchronized, and NTP service is configured and running..

Identify storage

Portworx pools the storage devices on your server and creates a global capacity for containers.

Important:
Back up any data on storage devices that will be pooled. Storage devices will be reformatted!

To view the storage devices on your server, use the lsblk command.

For example:

# lsblk
    NAME                      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    xvda                      202:0    0     8G  0 disk
    └─xvda1                   202:1    0     8G  0 part /
    xvdb                      202:16   0    64G  0 disk
    xvdc                      202:32   0    64G  0 disk

Note that devices without the partition are shown under the TYPE column as part. This example has two non-root storage devices (/dev/xvdb, /dev/xvdc) that are candidates for storage devices.

Identify the storage devices you will be allocating to PX. PX can run in a heterogeneous environment, so you can mix and match drives of different types. Different servers in the cluster can also have different drive configurations.

Install

PX runs as a container directly via OCI runC. This ensures that there are no cyclical dependencies between Docker and PX.

On each swarm node, perform the following steps to install PX.

Step 1: Install the PX OCI bundle

Portworx provides a Docker based installation utility to help deploy the PX OCI bundle. This bundle can be installed by running the following Docker container on your host system:

# Uncomment appropriate `REL` below to select desired Portworx release
REL=""          # DEFAULT portworx release
#REL="/1.4"     # 1.4 portworx release
#REL="/1.5"     # 1.5 portworx release
#REL="/1.6"     # 1.6 portworx release

latest_stable=$(curl -fsSL "https://install.portworx.com$REL/?type=dock&stork=false" | awk '/image: / {print $2}')

# Download OCI bits (reminder, you will still need to run `px-runc install ..` after this step)
sudo docker run --entrypoint /runc-entry-point.sh \
    --rm -i --privileged=true \
    -v /opt/pwx:/opt/pwx -v /etc/pwx:/etc/pwx \
    $latest_stable

Note:
Running the PX OCI bundle does not require Docker, but Docker will still be required to install the PX OCI bundle. If you do not have Docker installed on your target hosts, you can download this Docker package and extract it to a root tar ball and manually install the OCI bundle.

Step 2: Configure PX under runC

Now that you have downloaded and installed the PX OCI bundle, you can use the the px-runc install command from the bundle to configure systemd to start PX runC.

The px-runc command is a helper-tool that does the following:

  1. prepares the OCI directory for runC
  2. prepares the runC configuration for PX
  3. used by systemd to start the PX OCI bundle

Installation example:

#  Basic installation

sudo /opt/pwx/bin/px-runc install -c MY_CLUSTER_ID \
    -k etcd://myetc.company.com:2379 \
    -s /dev/xvdb -s /dev/xvdc -x swarm
Command-line arguments

Options

-c                        [REQUIRED] Specifies the cluster ID that this PX instance is to join
-k                        [REQUIRED] Points to your key value database, such as an etcd cluster or a consul cluster
-s                        [REQUIRED unless -a is used] Specifies the various drives that PX should use for storing the data
-e key=value              [OPTIONAL] Specify extra environment variables
-v <dir:dir[:shared,ro]>  [OPTIONAL] Specify extra mounts
-d <ethX>                 [OPTIONAL] Specify the data network interface
-m <ethX>                 [OPTIONAL] Specify the management network interface
-z                        [OPTIONAL] Instructs PX to run in zero storage mode
-f                        [OPTIONAL] Instructs PX to use an unmounted drive even if it has a filesystem on it
-a                        [OPTIONAL] Instructs PX to use any available, unused and unmounted drives
-A                        [OPTIONAL] Instructs PX to use any available, unused and unmounted drives or partitions
-j                        [OPTIONAL] Specifies a journal device for PX.  Specify a persistent drive like /dev/sdc or use auto (recommended)
-x <swarm|kubernetes>     [OPTIONAL] Specify scheduler being used in the environment
-r <portnumber>           [OPTIONAL] Specifies the portnumber from which PX will start consuming. Ex: 9001 means 9001-9020
  • additional PX-OCI -specific options:
-oci <dir>                [OPTIONAL] Specify OCI directory (default: /opt/pwx/oci)
-sysd <file>              [OPTIONAL] Specify SystemD service file (default: /etc/systemd/system/portworx.service)

KVDB options

-userpwd <user:passwd>    [OPTIONAL] Username and password for ETCD authentication
-ca <file>                [OPTIONAL] Specify location of CA file for ETCD authentication
-cert <file>              [OPTIONAL] Specify location of certificate for ETCD authentication
-key <file>               [OPTIONAL] Specify location of certificate key for ETCD authentication
-acltoken <token>         [OPTIONAL] ACL token value used for Consul authentication

Secrets options

-secret_type <aws|dcos|docker|k8s|kvdb|vault>   [OPTIONAL] Specify the secret type to be used by Portworx for cloudsnap and encryption features.
-cluster_secret_key <id>        [OPTIONAL] Specify the cluster wide secret key to be used when using AWS KMS or Vault for volume encryption.

Environment variables

PX_HTTP_PROXY         [OPTIONAL] If running behind an HTTP proxy, set the PX_HTTP_PROXY variables to your HTTP proxy.
PX_HTTPS_PROXY        [OPTIONAL] If running behind an HTTPS proxy, set the PX_HTTPS_PROXY variables to your HTTPS proxy.
PX_ENABLE_CACHE_FLUSH [OPTIONAL] Enable cache flush deamon. Set PX_ENABLE_CACHE_FLUSH=true.
PX_ENABLE_NFS         [OPTIONAL] Enable the PX NFS daemon. Set PX_ENABLE_NFS=true.

NOTE: Setting environment variables can be done using the -e option, during PX-OCI or PX Docker Container command line install (e.g. add -e VAR=VALUE option).

# Example PX-OCI config with extra "PX_ENABLE_CACHE_FLUSH" environment variable
$ sudo /opt/pwx/bin/px-runc install -e PX_ENABLE_CACHE_FLUSH=yes \
    -c MY_CLUSTER_ID -k etcd://myetc.company.com:2379 -s /dev/xvdb
Examples

Using etcd:

px-runc install -k etcd://my.company.com:2379 -c MY_CLUSTER_ID -s /dev/sdc -s /dev/sdb2 -x swarm
px-runc install -k etcd://70.0.1.65:2379 -c MY_CLUSTER_ID -s /dev/sdc -d enp0s8 -m enp0s8 -x swarm

Using consul:

px-runc install -k consul://my.company.com:8500 -c MY_CLUSTER_ID -s /dev/sdc -s /dev/sdb2 -x swarm
px-runc install -k consul://70.0.2.65:8500 -c MY_CLUSTER_ID -s /dev/sdc -d enp0s8 -m enp0s8 -x swarm
Modifying the PX configuration

After the initial installation, you can modify the PX configuration file at /etc/pwx/config.json (see details) and restart Portworx using systemctl restart portworx.

Step 3: Starting PX runC

Once you install the PX OCI bundle and systemd configuration from the steps above, you can start and control PX runC directly via systemd:

# Reload systemd configurations, enable and start Portworx service
sudo systemctl daemon-reload
sudo systemctl enable portworx
sudo systemctl start portworx

Note: If you have previously installed Portworx as a Docker container (as “legacy plugin system”, or v1 plugin), and already have PX-volumes allocated and in use by other Docker containers/applications, read instructions here

Adding Nodes

To add nodes to increase capacity and enable high availability, simply repeat these steps on other servers. As long as PX is started with the same cluster ID, they will form a cluster.

Access the pxctl CLI

After Portworx is running, you can create and delete storage volumes through the Docker volume commands or the pxctl command line tool.

With pxctl, you can also inspect volumes, the volume relationships with containers, and nodes. For more on using pxctl, see the CLI Reference.

To view the global storage capacity, run:

# sudo /opt/pwx/bin/pxctl status

The following sample output of pxctl status shows that the global capacity for Docker containers is 128 GB.

# /opt/pwx/bin/pxctl status
Status: PX is operational
Node ID: 0a0f1f22-374c-4082-8040-5528686b42be
	IP: 172.31.50.10
 	Local Storage Pool: 2 pools
	POOL	IO_PRIORITY	SIZE	USED	STATUS	ZONE	REGION
	0	LOW		64 GiB	1.1 GiB	Online	b	us-east-1
	1	LOW		128 GiB	1.1 GiB	Online	b	us-east-1
	Local Storage Devices: 2 devices
	Device	Path		Media Type		Size		Last-Scan
	0:1	/dev/xvdf	STORAGE_MEDIUM_SSD	64 GiB		10 Dec 16 20:07 UTC
	1:1	/dev/xvdi	STORAGE_MEDIUM_SSD	128 GiB		10 Dec 16 20:07 UTC
	total			-			192 GiB
Cluster Summary
	Cluster ID: 55f8a8c6-3883-4797-8c34-0cfe783d9890
	IP		ID					Used	Capacity	Status
	172.31.50.10	0a0f1f22-374c-4082-8040-5528686b42be	2.2 GiB	192 GiB		Online (This node)
Global Storage Pool
	Total Used    	:  2.2 GiB
	Total Capacity	:  192 GiB

Application Examples

Once you have Portworx up, take a look at an example of running stateful application with Portworx and Swarm!