Failover
Pod failover
Verify that your Cassandra cluster is formed of five nodes:
kubectl get pods -l "app=cassandra"
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 1h
cassandra-1 1/1 Running 0 10m
cassandra-2 1/1 Running 0 18h
cassandra-3 1/1 Running 0 17h
cassandra-4 1/1 Running 0 13h
Add data to Cassandra
-
Run the
bash
command on one of your Pods. The following example command runs thebash
command on thecassandra-2
Pod:kubectl exec -it cassandra-2 -- bash
-
Start
cqlsh
, the command line shell for interacting with Cassandra:cqlsh
Connected to TestCluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help. -
Enter the following example command to add data to a keyspace called
demodb
:CREATE KEYSPACE demodb WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 };
use demodb;
CREATE TABLE emp(emp_id int PRIMARY KEY, emp_name text, emp_city text, emp_sal varint,emp_phone varint);
INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES(123423445,'Steve', 'Denver', 5910234452, 50000); -
Run the
exit
command to terminatecqlsh
and return to the shell session. -
Display the list of nodes that host the data in your Cassandra ring based on its partition key:
nodetool getendpoints demodb emp 123423445
10.0.112.1
10.0.160.1 -
Terminate the shell session:
exit
-
Use the following command to list the nodes and the Pods they host:
kubectl get pods -l app=cassandra -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": .status.podIP}'
{
"name": "cassandra-0",
"hostname": "k8s-5",
"hostIP": "10.140.0.8",
"PodIP": "10.0.112.1"
}
{
"name": "cassandra-1",
"hostname": "k8s-0",
"hostIP": "10.140.0.3",
"PodIP": "10.0.160.1"
}
{
"name": "cassandra-2",
"hostname": "k8s-1",
"hostIP": "10.140.0.5",
"PodIP": "10.0.64.3"
}
{
"name": "cassandra-3",
"hostname": "k8s-3",
"hostIP": "10.140.0.6",
"PodIP": "10.0.240.1"
}
{
"name": "cassandra-4",
"hostname": "k8s-4",
"hostIP": "10.140.0.7",
"PodIP": "10.0.128.1"
}Note that the
k8s-0
node hosts thecassandra1
Pod.
Delete a Cassandra Pod
-
Cordon a node where one of the replicas resides. The Kubernetes stateful set will schedule the Pod to another node. The following
kubectl cordon
command cordons thek8s-0
node:kubectl cordon k8s-0
node "k8s-0" cordoned
-
Use the
kubectl delete pods
command to delete thecassandra-1
Pod:kubectl delete pods cassandra-1
pod "cassandra-1" deleted
-
The Kubernetes stateful set schedules the
cassandra-1
Pod on a different host. You can use thekubectl get pods -w
command to see where the Pod is in its lifecycle:kubectl get pods -w
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 1h
cassandra-1 0/1 ContainerCreating 0 1s
cassandra-2 1/1 Running 0 19h
cassandra-3 1/1 Running 0 17h
cassandra-4 1/1 Running 0 14h
cassandra-1 0/1 Running 0 4s
cassandra-1 1/1 Running 0 28s -
To see the node on which the Kubernetes stateful set schedules the
cassandra-1
Pod, enter the following command:kubectl get pods -l app=cassandra -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": status.podIP}'
{
"name": "cassandra-0",
"hostname": "k8s-5",
"hostIP": "10.140.0.8",
"PodIP": "10.0.112.1"
}
{
"name": "cassandra-1",
"hostname": "k8s-2",
"hostIP": "10.140.0.4",
"PodIP": "10.0.192.2"
}
{
"name": "cassandra-2",
"hostname": "k8s-1",
"hostIP": "10.140.0.5",
"PodIP": "10.0.64.3"
}
{
"name": "cassandra-3",
"hostname": "k8s-3",
"hostIP": "10.140.0.6",
"PodIP": "10.0.240.1"
}
{
"name": "cassandra-4",
"hostname": "k8s-4",
"hostIP": "10.140.0.7",
"PodIP": "10.0.128.1"
}Note that the
cassandra-1
Pod is now scheduled on thek8s-2
node. -
Verify that there is no data loss by entering the following command:
kubectl exec cassandra-1 -- cqlsh -e 'select * from demodb.emp'
emp_id | emp_city | emp_name | emp_phone | emp_sal
-----------+----------+----------+------------+---------
123423445 | Denver | Steve | 5910234452 | 50000
(1 rows)
Node failover
-
List the Pods in your cluster by entering the following command:
kubectl get pods -l app=cassandra -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": status.podIP}'
{
"name": "cassandra-0",
"hostname": "k8s-5",
"hostIP": "10.140.0.8",
"PodIP": "10.0.112.1"
}
{
"name": "cassandra-1",
"hostname": "k8s-2",
"hostIP": "10.140.0.4",
"PodIP": "10.0.192.2"
}
{
"name": "cassandra-2",
"hostname": "k8s-1",
"hostIP": "10.140.0.5",
"PodIP": "10.0.64.3"
}
{
"name": "cassandra-3",
"hostname": "k8s-3",
"hostIP": "10.140.0.6",
"PodIP": "10.0.240.1"
}
{
"name": "cassandra-4",
"hostname": "k8s-4",
"hostIP": "10.140.0.7",
"PodIP": "10.0.128.1"
}Note that Kubernetes scheduled the
cassandra-2
Pod on thek8s-1
node. -
Display the list of nodes and their labels:
kubectl get nodes --show-labels
NAME STATUS LABELS
k8s-0 Ready cassandra-data-cassandra-1=true,cassandra-data-cassandra-3=true
k8s-1 Ready cassandra-data-cassandra-1=true,cassandra-data-cassandra-4=true
k8s-2 Ready cassandra-data-cassandra-0=true,cassandra-data-cassandra-2=true
k8s-3 Ready cassandra-data-cassandra-3=true
k8s-4 Ready cassandra-data-cassandra-4=true
k8s-5 Ready
k8s-master Ready cassandra-data-cassandra-0=true,cassandra-data-cassandra-2=true
This example output is truncated for brevity.
-
Decommission the
k8s-1
Portworx node by following the steps in the Decommission a Node section. -
Decommission the
k8s-1
Kubernetes node by entering thekubectl delete node
command withk8s-1
as an argument:kubectl delete node k8s-1
-
List the Pods in your cluster by entering the following command:
kubectl get pods -l app=cassandra -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": .status.podIP}'
{
"name": "cassandra-0",
"hostname": "k8s-5",
"hostIP": "10.140.0.8",
"PodIP": "10.0.112.1"
}
{
"name": "cassandra-1",
"hostname": "k8s-2",
"hostIP": "10.140.0.4",
"PodIP": "10.0.192.2"
}
{
"name": "cassandra-2",
"hostname": "k8s-0",
"hostIP": "10.140.0.3",
"PodIP": "10.0.160.2"
}
{
"name": "cassandra-3",
"hostname": "k8s-3",
"hostIP": "10.140.0.6",
"PodIP": "10.0.240.1"
}
{
"name": "cassandra-4",
"hostname": "k8s-4",
"hostIP": "10.140.0.7",
"PodIP": "10.0.128.1"
}Note that the
cassandra-2
pod is scheduled on thek8s-0
node. -
Display the list of nodes and their labels:
kubectl get nodes --show-labels
NAME STATUS LABELS
k8s-0 Ready cassandra-data-cassandra-1=true,cassandra-data-cassandra-3=true
k8s-2 Ready cassandra-data-cassandra-0=true,cassandra-data-cassandra-2=true
k8s-3 Ready cassandra-data-cassandra-3=true
k8s-4 Ready cassandra-data-cassandra-4=true
k8s-5 Ready
k8s-master Ready cassandra-data-cassandra-0=true,cassandra-data-cassandra-2=true