Migrate from DaemonSet to Operator in GCP Anthos
This page describes how to migrate your Portworx cluster from a DaemonSet installation to a Portworx Operator installation.
Prerequisites
To migrate, you must meet the following prerequisites:
- Kubernetes version 1.16 or newer
- Portworx installed with one DaemonSet; the scenario of two or more DaemonSet installations on one Kubernetes cluster is not supported
Migrate to Operator
Install Portworx Operator
Install the Portworx Operator version 1.9.0 or newer using one of the following methods:
Install by spec generator
To install using the spec generator, run the following commands:
curl -o operator.yaml https://install.portworx.com/?comp=pxoperator
kubectl apply -f operator.yaml
Wait for the StorageCluster to be created
If you have Portworx DaemonSet installed, the Operator will automatically detect that on startup. The Operator will then create an equivalent StorageCluster object.
Add cloud provider annotation to StorageCluster
Run the appropriate command for your cloud provider from the following list:
- Google Kubernetes Engine (GKE)
kubectl -n kube-system annotate storagecluster --all portworx.io/is-gke="true"
Perform a dry run (optional)
Portworx by Pure Storage recommends running a dry run and reviewing the specs:
-
Find the Portworx Operator pod:
kubectl -n kube-system get pods -l name=portworx-operator
-
Execute the dry run tool in the Portworx Operator container:
kubectl -n kube-system exec -it <portworx-operator-pod> -- /dryrun
If the dry run is successful, the end of the output will say Operator migration dry-run finished successfully
.
To review the specs that the Operator will deploy after migration, especially if you have any custom changes to your DaemonSet deployment, run the following commands:
-
List all of the specs that the Operator will deploy after migration:
kubectl -n kube-system exec -it <portworx-operator-pod> -- ls /tmp/portworxSpecs
-
Preview the Portworx pod spec after migration:
kubectl -n kube-system exec -it <portworx-operator-pod> -- cat /tmp/portworxSpecs/Pod.portworx.yaml
Migration using OnDelete update strategy (optional)
If you are not using OnDelete
for your StorageCluster update strategy, proceed to the next step.
If the StorageCluster update strategy is OnDelete
(which means that DaemonSet has the OnDelete
update strategy or the StorageCluster was updated manually to include this strategy), you will need to manually migrate Portworx pods on the nodes. Migration will be blocked until all Portworx nodes are migrated. After all Portworx nodes are migrated, Operator will migrate the components automatically.
-
Run the following command to approve the migration:
kubectl -n kube-system annotate storagecluster --all --overwrite portworx.io/migration-approved='true'
-
Change the migration label to
Starting
on the KVDB nodes:kubectl label node <kvdb-node> portworx.io/daemonset-migration=Starting --overwrite
A KVDB node should be migrated one at a time, and wait until all KVDB nodes are migrated and pods are created:
-
Change the migration label to
Starting
on nodes to trigger migration on a Portworx node:kubectl label node <portworx-node> portworx.io/daemonset-migration=Starting --overwrite
Operator starts migrating the labeled nodes at once.
- If the update strategy is changed from
OnDelete
toRollingUpdate
during the nodes migration, the Operator will pick up the Portworx node in progress and continue the migration process automatically. - If the update strategy is changed from
RollingUpdate
toOnDelete
during the nodes migration, the procedure will be blocked after the current node migration is done, then the procedure will wait for manual approval.
You can skip the next section and proceed to this section to verify the status of your migration.
Approve the migration
To approve the migration, run the following command:
kubectl -n kube-system annotate storagecluster --all --overwrite portworx.io/migration-approved='true'
Check migration status
Describe the StorageCluster using the following command:
kubectl -n kube-system describe storagecluster
If the migration completes successfully, you will see the event Migration completed successfully
. If migration fails, there is corresponding event about the failure.
Rollback to DaemonSet
If the migration from DaemonSet to the Operator fails for some reason and you want to rollback to the DaemonSet installation, perform the following steps.
Save DaemonSet backup to a file
Save the DaemonSet backup, which is automatically generated by the Operator, to a YAML file. Replace kube-system
with the DaemonSet namespace in the following command:
kubectl -n kube-system describe configmap px-daemonset-backup | grep -v Events | tail -n +10 > backup.yaml
Remove Operator installation
To remove the Operator installation, perform the following steps:
-
Delete the Operator installation by deleting the StorageCluster CRD:
kubectl -n kube-system delete storagecluster --all
-
Delete Operator with the following command:
kubectl -n kube-system delete deployment portworx-operator
-
Remove the migration labels from the nodes:
kubectl label node --all --overwrite portworx.io/daemonset-migration-
Apply the DaemonSet backup
Install with DaemonSet backup using the following command:
kubectl apply -f backup.yaml
Troubleshooting
One or more Portworx nodes are already down for some known reason
If Portworx is down on a node, migration will get stuck on that node as it tries to ensure that Portworx is online before it can proceed to the next node. If you know that the Portworx node is expected to be down, you can skip that node and let the migration continue to the next node. Add the following label on the node on which Portworx is down:
kubectl label node <node-name> portworx.io/daemonset-migration=Skip --overwrite
Grafana isn’t working after migration
If the Grafana pod is running with the Prometheus endpoint http://prometheus:9090
, then the Grafana portal won’t work after migrating to Operator from DaemonSet. Perform the following modifications:
-
Run the following command:
kubectl edit configmap grafana-source-config -n kube-system
-
Replace
url: http://prometheus:9090
with the following:url: http://px-prometheus:9090
-
Bounce the Grafana pod by deleting the existing Grafana pod.
Prometheus service name change
Prometheus should still work after migration, but the name of the service will change to px-prometheus
, whereas the default service name in the DaemonSet installation is prometheus
. To verify this, run the following command:
kubectl get service -n kube-system
px-prometheus ClusterIP X.X.X.0 <none> 9090/TCP