Skip to main content
Version: 3.2

ScyllaDB on Kubernetes and Portworx

ScyllaDB is a NoSQL database that aims to provide low latency and high throughput. Portworx is a software-defined storage overlay that allows you to run highly-available stateful applications. This article shows how you can create and run a ScyllaDB cluster on Kubernetes, which stores data on Portworx volumes.

Prerequisites​

  • You must have a Kubernetes cluster with a minimum of 3 worker nodes.
  • Portworx is installed on your Kubernetes cluster. For more details on how to install Portworx, refer to the instructions from the Portworx on Kubernetes page.

Create a StorageClass for volume provisioning​

  1. Use the following command to list the nodes in your cluster:

    kubectl get nodes -o wide
    NAME                           STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
    ravi-blr-dev-dour-shoulder-0   Ready    master   103d   v1.14.1   X.X.X.119   <none>        CentOS Linux 7 (Core)   3.10.0-862.3.2.el7.x86_64   docker://18.9.6
    ravi-blr-dev-dour-shoulder-1   Ready    <none>   16d    v1.14.1   X.X.X.82    <none>        CentOS Linux 7 (Core)   3.10.0-862.3.2.el7.x86_64   docker://18.9.6
    ravi-blr-dev-dour-shoulder-2   Ready    <none>   103d   v1.14.1   X.X.X.118   <none>        CentOS Linux 7 (Core)   3.10.0-862.3.2.el7.x86_64   docker://18.9.6
    ravi-blr-dev-dour-shoulder-3   Ready    <none>   103d   v1.14.1   X.X.X.120   <none>        CentOS Linux 7 (Core)   3.10.0-862.3.2.el7.x86_64   docker://18.9.6

  2. Define the following portworx-sc.yaml StorageClass:

    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: scylla-ssd
    provisioner: pxd.portworx.com
    parameters:
      repl: "2"
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    allowVolumeExpansion: true

  3. Apply the StorageClass configuration:

    kubectl apply -f portworx-sc.yml

ScyllaDB installation​

  1. Create a file called scylla-configmap.yaml with the following content:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: scylla
    data:
      ready-probe.sh: |
        #!/bin/bash

        if [[ $(nodetool status | grep $POD_IP) == *"UN"* ]]; then
          if [[ $DEBUG ]]; then
            echo "UN";
          fi
          exit 0;
        else
          if [[ $DEBUG ]]; then
            echo "Not Up";
          fi
          exit 1;
        fi

  2. Apply the ConfigMap:

    kubectl apply -f scylla-configmap.yaml

  3. Create a file called scylla-service.yaml with the following content:

    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: scylla
      name: scylla
    spec:
      clusterIP: None
      ports:
        - port: 9042
      selector:
        app: scylla

  4. Apply the Service:

    kubectl apply -f scylla-service.yaml

  5. The spec below creates StatefulSet for ScyllaDB with 3 replicas and uses the Stork scheduler to place pods to closer to where their data is located. Create a file called scylla-statefulset.yaml with the following content:

    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
    name: scylla
    labels:
      app: scylla
    spec:
    serviceName: scylla
    replicas: 3
    selector:
      matchLabels:
        app: scylla
    template:
      metadata:
        labels:
          app: scylla
      spec:
        schedulerName: stork
        containers:
          - name: scylla
            image: scylladb/scylla:2.0.0
            imagePullPolicy: IfNotPresent
            args: ["--seeds", "scylla-0.scylla.default.svc.cluster.local"]
            ports:
              - containerPort: 7000
                name: intra-node
              - containerPort: 7001
                name: tls-intra-node
              - containerPort: 7199
                name: jmx
              - containerPort: 9042
                name: cql
            resources:
              limits:
                cpu: 500m
                memory: 1Gi
              requests:
                cpu: 500m
                memory: 1Gi
            securityContext:
              capabilities:
                add:
                  - IPC_LOCK
            lifecycle:
              preStop:
                exec:
                  command: ["/bin/sh", "-c", "PID=$(pidof scylla) && kill $PID && while ps -p $PID > /dev/null; do sleep 1; done"]
            env:
              - name: POD_IP
                valueFrom:
                  fieldRef:
                    fieldPath: status.podIP

            readinessProbe:
              exec:
                command:
                  - /bin/bash
                  - -c
                  - exec
                  - /opt/ready-probe.sh
              initialDelaySeconds: 15
              timeoutSeconds: 5
            volumeMounts:
              - name: scylla-data
                mountPath: /var/lib/scylla
              - name: scylla-ready-probe
                mountPath: /opt/ready-probe.sh
                subPath: ready-probe.sh
        volumes:
          - name: scylla-ready-probe
            configMap:
              name: scylla
    volumeClaimTemplates:
      - metadata:
          name: scylla-data
          annotations:
            volume.beta.kubernetes.io/storage-class: scylla-ssd
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 60Gi

  6. Apply the scylla-statefulset.yaml StatefulSet:

    kubectl apply scylla-statefulset.yaml

Verify ScyllaDB installation​

  1. Enter the kubectl get pvc command to verify that the PVCs are bound to a volume using the storage class. The PVC status shows as Bound if the operation succeeded:

    kubectl get pvc
    NAME                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    scylla-data-scylla-0   Bound    pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29886e3e   60Gi       RWO            scylla-ssd     3h
    scylla-data-scylla-1   Bound    pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29886e3e   60Gi       RWO            scylla-ssd     3h
    scylla-data-scylla-2   Bound    pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29886e3e   60Gi       RWO            scylla-ssd     3h

  2. Enter the kubectl get pods command to verify that the ScyllaDB pods have deployed successfully. The pod status shows as Running if the operation succeeded:

    kubectl get pods
    NAME       READY   STATUS    RESTARTS   AGE
    scylla-0   1/1     Running   0          3h
    scylla-1   1/1     Running   0          3h
    scylla-2   1/1     Running   0          3h

  3. Run the nodetool status command in the scylla-0 pod to verify that the ScyllaDB cluster was created:

    kubectl exec scylla-0 -- nodetool status
    Datacenter: datacenter1
    =======================
    Status=Up/Down
    |/ State=Normal/Leaving/Joining/Moving
    --  Address        Load       Tokens       Owns (effective)  Host ID                               Rack
    UN  10.233.127.67  356.04 KB  256          61.5%             xxxxxxxx-xxxx-xxxx-xxxx-d9b928f1fc4a  rack1
    UN  10.233.76.19   351.15 KB  256          70.5%             xxxxxxxx-xxxx-xxxx-xxxx-c897c7441895  rack1
    UN  10.233.121.53  359.24 KB  256          68.0%             xxxxxxxx-xxxx-xxxx-xxxx-640ce9ec9f6e  rack1

    Note the pods placement and the hosts on which they are scheduled

  4. Enter the kubectl get pods command, filtering the output using the jq command to display the following:

  • Pod name

  • Hostname

  • Host IP

  • Pod IP

    kubectl get pods -l app=scylla -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": .status.podIP}'
    {
      "name": "scylla-0",
      "hostname": "ravi-blr-dev-dour-shoulder-3",
      "hostIP": "X.X.X.120",
      "PodIP": "10.233.121.53"
    }
    {
      "name": "scylla-1",
      "hostname": "ravi-blr-dev-dour-shoulder-1",
      "hostIP": "X.X.X.82",
      "PodIP": "10.233.76.19"
    }
    {
      "name": "scylla-2",
      "hostname": "ravi-blr-dev-dour-shoulder-2",
      "hostIP": "X.X.X.118",
      "PodIP": "10.233.127.67"
    }

  1. Enter the ssh command to open a shell session into one of your nodes:

    ssh X.X.X.120

  2. Enter the pxctl volume list command to list your volume IDs. Save one of the node IDs for future reference:

    pxctl volume list
    ID			NAME						SIZE	HA	SHARED	ENCRYPTED	IO_PRIORITY	STATUS			SNAP-ENABLED
    242236313329814877	pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29886e3e	60 GiB	2	no	no		LOW		up - attached on X.X.X.120	no
    702215287887827398	pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29886e3e	60 GiB	2	no	no		LOW		up - attached on X.X.X.82	no
    685261507172158119	pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29886e3e	60 GiB	2	no	no		LOW		up - attached on X.X.X.118	no

  3. Enter the pxctl volume inspect command to examine your volume. In the example output below, the Portworx volume contains 2 replica sets and is attached to node 3:

    pxctl volume inspect 242236313329814877
    Volume	:  242236313329814877
      Name            	 :  pvc-xxxxxxxx-xxxx-xxxx-xxxx-000c29886e3e
      Size            	 :  60 GiB
      Format          	 :  ext4
      HA              	 :  2
      IO Priority     	 :  LOW
      Creation time   	 :  Oct 4 05:14:10 UTC 2019
      Shared          	 :  no
      Status          	 :  up
      State           	 :  Attached: xxxxxxxx-xxxx-xxxx-xxxx-fdb2c8379541 (X.X.X.120)
      Device Path     	 :  /dev/pxd/pxd242236313329814877
      Labels          	 :  pvc=scylla-data-scylla-0,repl=2,namespace=default
      Reads           	 :  887
      Reads MS        	 :  553
      Bytes Read      	 :  4526080
      Writes          	 :  4086
      Writes MS       	 :  27087
      Bytes Written   	 :  1032036352
      IOs in progress 	 :  0
      Bytes used      	 :  24 MiB
      Replica sets on nodes:
        Set 0
          Node 		 : X.X.X.120 (Pool 2)
          Node 		 : X.X.X.82 (Pool 2)
      Replication Status	 :  Up
      Volume consumers	 :
        - Name           : scylla-0 (xxxxxxxx-xxxx-xxxx-xxxx-000c29886e3e) (Pod)
          Namespace      : default
          Running on     : ravi-blr-dev-dour-shoulder-3
          Controlled by  : scylla (StatefulSet)

Failover​

Once you've created a ScyllaDB cluster on Kubernetes and Portworx, you can test how the cluster reacts to a failure.

Pod failover​

The steps in this exercise simulate a pod failure and demonstrate Portworx and Kubernetes' ability to recover from that failure.

  1. List the pods on which ScyllaDB is running:

    kubectl get pods -l "app=scylla"
    NAME       READY   STATUS    RESTARTS   AGE
    scylla-0   1/1     Running   0          4h
    scylla-1   1/1     Running   0          4h
    scylla-2   1/1     Running   0          4h

  2. Enter the following kubectl exec command to open a bash session with the worker node on which ScyllaDB is running:

    kubectl exec -it scylla-0 -- bash

  3. Enter the cqlsh command to open a Cassandra shell session:

    cqlsh
    Connected to Test Cluster at 10.233.121.53:9042.
    [cqlsh 5.0.1 | Cassandra 3.0.8 | CQL spec 3.3.1 | Native protocol v4]
    Use HELP for help.
    cqlsh>

  4. Enter the following CREATE KEYSPACE statement to create a keyspace named demodb:

    CREATE KEYSPACE demodb WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 };

  5. Enter the following commands to switch to the demodb keyspace and create a table called emp within it:

    cqlsh> use demodb;
    cqlsh:demodb> CREATE TABLE emp(emp_id int PRIMARY KEY, emp_name text, emp_city text, emp_sal varint,emp_phone varint);

  6. Enter the following INSERT INTO statement to insert a record into the emp table:

    cqlsh:demodb> INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES(123423445,'Steve', 'Denver', 5910234452, 50000);

  7. Exit the Cassandra shell session:

    cqlsh> exit

  8. Enter the following nodetool getendpoints command to list the IP addresses of the nodes which are also hosting ScyllaDB information based on the partition key:

    nodetool getendpoints demodb emp 123423445
    10.233.76.19
    10.233.127.67

  9. Enter the following kubectl get pods command to crosscheck the pod IP addresses with the IP addresses you just listed:

    kubectl get pods -l app=scylla -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": .status.podIP}'
    {
      "name": "scylla-0",
      "hostname": "ravi-blr-dev-dour-shoulder-3",
      "hostIP": "X.X.X.120",
      "PodIP": "10.233.121.53"
    }
    {
      "name": "scylla-1",
      "hostname": "ravi-blr-dev-dour-shoulder-1",
      "hostIP": "X.X.X.82",
      "PodIP": "10.233.76.19"
    }
    {
      "name": "scylla-2",
      "hostname": "ravi-blr-dev-dour-shoulder-2",
      "hostIP": "X.X.X.118",
      "PodIP": "10.233.127.67"
    }

  10. Enter the following kubectl cordon command to mark the scylla-1 node as unschedulable:

    kubectl cordon ravi-blr-dev-dour-shoulder-1

  11. Enter the following kubectl delete command to delete the syclla-1 pod:

    kubectl delete pods scylla-1

  12. Enter the following kubectl get pods command to see if Kubernetes scheduled the syclla-1 pod to a different node:

    kubectl get pods -o wide

    NAME       READY   STATUS    RESTARTS   AGE   IP              NODE                           NOMINATED NODE   READINESS GATES
    scylla-0   1/1     Running   0          4h    10.233.121.53   ravi-blr-dev-dour-shoulder-3   <none>           <none>
    scylla-1   1/1     Running   0          25s   10.233.127.68   ravi-blr-dev-dour-shoulder-2   <none>           <none>
    scylla-2   1/1     Running   0          4h    10.233.127.67   ravi-blr-dev-dour-shoulder-2   <none>           <none>

    Note that the scyla-1 pod is now scheduled to the ravi-blr-dev-dour-shoulder-2 node.

  13. Enter the kubectl get pods command, filtering the output using the jq command to display the following:

  • Pod name
  • Host name
  • Host IP
  • Pod IP
kubectl get pods -l app=scylla -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": .status.podIP}'
{
  "name": "scylla-0",
  "hostname": "ravi-blr-dev-dour-shoulder-3",
  "hostIP": "X.X.X.120",
  "PodIP": "10.233.121.53"
}
{
  "name": "scylla-1",
  "hostname": "ravi-blr-dev-dour-shoulder-2",
  "hostIP": "X.X.X.118",
  "PodIP": "10.233.127.68"
}
{
  "name": "scylla-2",
  "hostname": "ravi-blr-dev-dour-shoulder-2",
  "hostIP": "X.X.X.118",
  "PodIP": "10.233.127.67"
}

  1. Enter the following SELECT statement to read the columns from the demodb.emp table:

    kubectl exec scylla-1 -- cqlsh -e 'select * from demodb.emp'
     emp_id    | emp_city | emp_name | emp_phone  | emp_sal
    -----------+----------+----------+------------+---------
     123423445 |   Denver |    Steve | 5910234452 |   50000

    (1 rows)
Last updated on