Migrate from DaemonSet to Operator installation
This page describes how to migrate your Portworx cluster from a DaemonSet installation to a Portworx Operator installation.
Prerequisites
To migrate, you must meet the following prerequisites:
- Kubernetes version 1.16 or newer
- Portworx installed with one DaemonSet; the scenario of two or more DaemonSet installations on one Kubernetes cluster is not supported
Migrate to Operator
Install Portworx Operator
Install the Portworx Operator version 1.9.0 or newer using one of the following methods:
Install by spec generator
To install using the spec generator, run the following commands:
curl -o operator.yaml https://install.portworx.com/?comp=pxoperator
oc apply -f operator.yaml
Wait for the StorageCluster to be created
If you have Portworx DaemonSet installed, the Operator will automatically detect that on startup. The Operator will then create an equivalent StorageCluster object.
Add OCP annotation to StorageCluster
Run the following command for OpneShift cluster:
oc -n kube-system annotate storagecluster --all portworx.io/is-openshift="true"
Perform a dry run (optional)
Portworx by Pure Storage recommends running a dry run and reviewing the specs:Find the Portworx Operator pod:
oc -n kube-system get pods -l name=portworx-operator
Execute the dry run tool in the Portworx Operator container:
oc -n kube-system exec -it <portworx-operator-pod> -- /dryrun
If the dry run is successful, the end of the output will say Operator migration dry-run finished successfully
.
To review the specs that the Operator will deploy after migration, especially if you have any custom changes to your DaemonSet deployment, run the following commands:
List all of the specs that the Operator will deploy after migration:
oc -n kube-system exec -it <portworx-operator-pod> -- ls /tmp/portworxSpecs
Preview the Portworx pod spec after migration:
oc -n kube-system exec -it <portworx-operator-pod> -- cat /tmp/portworxSpecs/Pod.portworx.yaml
Migration using OnDelete update strategy (optional)
If you are not using OnDelete
for your StorageCluster update strategy, proceed to the next step.
If the StorageCluster update strategy is OnDelete
(which means that DaemonSet has the OnDelete
update strategy or the StorageCluster was updated manually to include this strategy), you will need to manually migrate Portworx pods on the nodes. Migration will be blocked until all Portworx nodes are migrated. After all Portworx nodes are migrated, Operator will migrate the components automatically.
Run the following command to approve the migration:
oc -n kube-system annotate storagecluster --all --overwrite portworx.io/migration-approved='true'
Change the migration label to
Starting
on the KVDB nodes:oc label node <kvdb-node> portworx.io/daemonset-migration=Starting --overwrite
A KVDB node should be migrated one at a time, and wait until all KVDB nodes are migrated and pods are created:
Change the migration label to
Starting
on nodes to trigger migration on a Portworx node:oc label node <portworx-node> portworx.io/daemonset-migration=Starting --overwrite
Operator starts migrating the labeled nodes at once.
- If the update strategy is changed from
OnDelete
toRollingUpdate
during the nodes migration, the Operator will pick up the Portworx node in progress and continue the migration process automatically. - If the update strategy is changed from
RollingUpdate
toOnDelete
during the nodes migration, the procedure will be blocked after the current node migration is done, then the procedure will wait for manual approval.
You can skip the next section and proceed to this section to verify the status of your migration.
Approve the migration
To approve the migration, run the following command:
oc -n kube-system annotate storagecluster --all --overwrite portworx.io/migration-approved='true'
Check migration status
Describe the StorageCluster using the following command:
oc -n kube-system describe storagecluster
If the migration completes successfully, you will see the event Migration completed successfully
. If migration fails, there is corresponding event about the failure.
Rollback to DaemonSet
If the migration from DaemonSet to the Operator fails for some reason and you want to rollback to the DaemonSet installation, perform the following steps.
Save DaemonSet backup to a file
Save the DaemonSet backup, which is automatically generated by the Operator, to a YAML file. Replace kube-system
with the DaemonSet namespace in the following command:
oc -n kube-system describe configmap px-daemonset-backup | grep -v Events | tail -n +10 > backup.yaml
Remove Operator installation
To remove the Operator installation, perform the following steps:
Delete the Operator installation by deleting the StorageCluster CRD:
oc -n kube-system delete storagecluster --all
Delete Operator with the following command:
oc -n kube-system delete deployment portworx-operator
Remove the migration labels from the nodes:
oc label node --all --overwrite portworx.io/daemonset-migration-
Apply the DaemonSet backup
Install with DaemonSet backup using the following command:
oc apply -f backup.yaml
Troubleshooting
One or more Portworx nodes are already down for some known reason
If Portworx is down on a node, migration will get stuck on that node as it tries to ensure that Portworx is online before it can proceed to the next node. If you know that the Portworx node is expected to be down, you can skip that node and let the migration continue to the next node. Add the following label on the node on which Portworx is down:
oc label node <node-name> portworx.io/daemonset-migration=Skip --overwrite
Grafana isn’t working after migration
If the Grafana pod is running with the Prometheus endpoint http://prometheus:9090
, then the Grafana portal won’t work after migrating to Operator from DaemonSet. Perform the following modifications:
Run the following command:
oc edit configmap grafana-source-config -n kube-system
Replace
url: http://prometheus:9090
with the following:url: http://px-prometheus:9090
Bounce the Grafana pod by deleting the existing Grafana pod.
Prometheus service name change
Prometheus should still work after migration, but the name of the service will change to px-prometheus
, whereas the default service name in the DaemonSet installation is prometheus
. To verify this, run the following command:
oc get service -n kube-system
px-prometheus ClusterIP X.X.X.0 <none> 9090/TCP