Configure 3DSnaps in ROSA
Summary and Key concepts
Summary:
This article explains how to configure 3DSnaps in Portworx, allowing users to run pre- and post-snapshot rules on application pods. These rules can quiesce the application (e.g., flush tables or lock databases) before a snapshot and resume operations after the snapshot. The process involves creating Stork rules (which define the actions and pods to be affected) and then referencing those rules in VolumeSnapshot or GroupVolumeSnapshot objects. Several example rules are provided for applications like MySQL, MongoDB, and Cassandra, showing how to perform pre- and post-snapshot operations using commands. The article also covers how to verify and restore snapshots.
Kubernetes Concepts:
- VolumeSnapshot: Allows users to take snapshots of PersistentVolumes, with pre- and post-snapshot rules applied for quiescing and resuming applications.
- PersistentVolumeClaim (PVC): The snapshots apply to the volumes managed by PVCs in Kubernetes.
- Custom Resource Definition (CRD): The Stork
Rule
CRD defines the pre- and post-snapshot operations in this workflow.
Portworx Concepts:
-
Stork: A tool used to manage storage operations, including executing pre- and post-snapshot rules during snapshot creation.
-
Group Snapshots: A feature allowing users to take a snapshot of multiple volumes at once, typically used with pre- and post-execution rules.
For each of the snapshot types, Portworx supports specifying pre and post rules that are run on the application pods using the volumes being snapshotted. This allows users to quiesce the applications before the snapshot is taken and resume I/O after the snapshot is taken.
The high level workflow for configuring 3DSnaps involves creating rules and later on referencing the rules when creating the snapshots.
Step 1: Create Rules
A Stork Rule
is a Custom Resource Definition (CRD) that allows to define actions that get performed on pods matching selectors. Below are the supported fields:
- podSelector: The actions will get executed on pods that only match the label selectors given here.
- actions: This contains a list of actions to be performed. Below are supported fields under actions:
- type: The type of action to run. Only type command is supported.
- background: If true, the action will run in background and will be terminated by Stork after the snapshot has been initiated. If false, the action will first complete and then the snapshot will get initiated.
- If background is set to true, add
${WAIT_CMD}
as shown in the examples below. This is a placeholder and Stork will replace it with an appropriate command to wait for the command is done.
- If background is set to true, add
- value: This is the actual action content. For example, the command to run.
- runInSinglePod: If true, the action will be run on a single pod that matches the selectors.
- container: specifies the container in which Stork will execute the action.
Step 2: Create snapshots that reference the rules
Once you have the rules applied in your cluster, you can reference them in the VolumeSnapshot
or GroupVolumeSnapshot
for individual and group snapshots respectively.
VolumeSnapshots
Use following annotations on the VolumeSnapshot to specify pre and post rules.
- stork.rule/pre-snapshot: Stork will execute the rule which is given in the value of this annotation before taking the snapshot.
- stork.rule/post-snapshot: Stork will execute the rule which is given in the value of this annotation after taking the snapshot.
GroupVolumeSnapshots
Use following fields in the GroupVolumeSnapshot spec to specify pre and post rules.
- preExecRule: Stork will execute the rule which is given in the value of this annotation before taking the group snapshot.
- postExecRule: Stork will execute the rule which is given in the value of this annotation after taking the snapshot.
Examples
This section covers examples of creating 3DSnapshots for various applications.
Specify the container in which Stork will execute the action
The following example rule configures Stork to execute the action in a container named <my_container>
:
apiVersion: stork.libopenstorage.org/v1alpha1
kind: Rule
metadata:
name: mysql-pre-backup-rule
namespace: mysql-1-pvc-mysql
annotations:
"stork/cmdexecutor-image": "openstorage/cmdexecutor:latest"
rules:
- podSelector:
app: mysql
container: <my_container>
Hello world
Below rule is a generic example on how to run an echo command on a single pod that matches the label selector app=foo.
apiVersion: stork.libopenstorage.org/v1alpha1
kind: Rule
metadata:
name: px-hello-world-rule
rules:
- podSelector:
app: foo
actions:
- type: command
value: echo "hello world"
runInSinglePod: true
Mysql
Pre-snapshot rule
Below rule will flush tables on all mysql pods that match label app=mysql and take a read lock on the tables.
apiVersion: stork.libopenstorage.org/v1alpha1
kind: Rule
metadata:
name: px-presnap-rule
rules:
- podSelector:
app: mysql
actions:
- type: command
background: true
# this command will flush tables with read lock
value: mysql --user=root --password=$MYSQL_ROOT_PASSWORD -Bse 'flush tables with read lock;system ${WAIT_CMD};'
Snapshot
Creating the below VolumeSnapshot will do the following:
-
Stork will run the px-presnap-rule rule on the pod that's using the mysql-data PVC.
-
Once the rule is executed, Stork will take a snapshot of the mysql-data PVC.
-
After the snapshot has been triggered, Stork will terminate any background actions that may exist in the rule px-presnap-rule.
apiVersion: volumesnapshot.external-storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: mysql-3d-snapshot
annotations:
stork.rule/pre-snapshot: px-presnap-rule
spec:
persistentVolumeClaimName: mysql-data
MongoDB
Pre-snapshot rule
Below pre-snapshot rule forces the mongod to flush all pending write operations to disk and locks the entire mongod instance to prevent additional writes until the user releases the lock with a corresponding db.fsyncUnlock() command. (See reference)
apiVersion: stork.libopenstorage.org/v1alpha1
kind: Rule
metadata:
name: px-mongodb-presnap-rule
rules:
- podSelector:
app: px-mongo-mongodb
actions:
- type: command
value: mongo --eval "printjson(db.fsyncLock())"
Post-snapshot rule
Below post-snapshot rule reduces the lock taken by db.fsyncLock() on a mongod instance by 1. (See reference)
apiVersion: stork.libopenstorage.org/v1alpha1
kind: Rule
metadata:
name: px-mongodb-postsnap-rule
rules:
- podSelector:
app: px-mongo-mongodb
actions:
- type: command
value: mongo --eval "printjson(db.fsyncUnlock())"
Snapshot
Creating the below VolumeSnapshot will do the following:
-
Stork will run the px-mongodb-presnap-rule rule on pod using the px-mongo-pvc PVC.
-
Once the rule is executed, Stork will take a snapshot of the px-mongo-pvc PVC.
-
After the snapshot has been triggered, Stork will run the px-mongodb-postsnap-rule rule on pod using the px-mongo-pvc PVC.
apiVersion: volumesnapshot.external-storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: mongodb-3d-snapshot
annotations:
stork.rule/pre-snapshot: px-mongodb-presnap-rule
stork.rule/post-snapshot: px-mongodb-postsnap-rule
spec:
persistentVolumeClaimName: px-mongo-pvc
Cassandra
Pre-snapshot rule
Below rule flushes the tables from the memtable on all cassandra pods.
apiVersion: stork.libopenstorage.org/v1alpha1
kind: Rule
metadata:
name: cassandra-presnap-rule
rules:
- podSelector:
app: cassandra
actions:
- type: command
value: nodetool flush
Snapshot
With this snapshot, Stork will run the cassandra-presnap-rule rule on all pods that are using PVCs that match labels app=cassandra. Hence this will be a group snapshot.
apiVersion: stork.libopenstorage.org/v1alpha1
kind: GroupVolumeSnapshot
metadata:
name: cassandra-group-snapshot
spec:
preExecRule: cassandra-presnap-rule
pvcSelector:
matchLabels:
app: cassandra
References
- For details about how you can restore a snapshot to a new PVC or the original PVC, see the Restore snapshots section.
- For details about how you can create PVCs from group snapshots, see the Creating PVCs from group snapshots section.