Back up Elasticsearch on Kubernetes

You can use the instructions on this page to create pre and post backup rules with Portworx Backup to take application-consistent backups for Elasticsearch on Kubernetes in production.

The Elasticsearch data directory is used to prevent permanent data loss and is typically located at /user/share/elasticsearch/data inside the Kubernetes pod. This will also enable Portworx Backup to back up and restore the data stored in this location.

Before using this guide, make sure you configure PVCs for elasticsearch-data. Use the following file as an example:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 7.7.0
  nodeSets:
  - name: default
    count: 3
    podTemplate:
      metadata:
        labels:
          appname: "elastisearch-app"
      spec:
        containers:
          - name: elasticsearch
            volumeMounts:
            - name: elasticsearch-backups
              mountPath: /usr/share/elasticsearch/backups
        volumes:
        - name: elasticsearch-backups
          persistentVolumeClaim:
            claimName: elasticsearch-backups
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: elastic-pwx-storage-class
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
      path.repo: ["/usr/share/elasticsearch/backups"]

Elasticsearch can also create a snapshot repository to store index snapshots produced by the internal snapshot and restore API. For this, the above template uses a Portworx shared volume and named elasticsearch-backups to create a shared file-system across all Elasticsearch nodes to be used to store the index snapshots.

Installation

Prerequisites

note

• The template above cannot be used alone. Please follow prerequisites from the following Elastic on kubernetes operations guide.

• The rules below will use a username elastic and password specific to the environment. You will need to modify the rule to use your username and password for your environment.

• You should mount a PVC to the Elasticsearch data directory. This is typically located at /user/share/elasticsearch/data.

note

If your Elasticsearch version is 7.7 and lower, you need to create pre-exec and post-exec rules to back up Elasticsearch data.

Create rules for Elasticsearch

Create rules for Elasticsearch that runs both before and after the backup operation runs:

Create a pre-exec backup rule for Elasticsearch

For the pre-backup rule you will create a rule that performs multiple actions.

• Freeze the index
• Flush all indexes in Elasticsearch
• Create an Elasticsearch index snapshot of all indexes

Create the rule:

1. From the home page, navigate to Settings > Rules >Add New.

2. In the Add Rule window, provide the following details:

   • Rule name: add a name for your backup rule

   • Pod Selector: add the following app label appname=elasticsearch-app

   • Container: ignore this field

   • Action: add the following action

     curl -X POST -u "elastic:<password>" -k "https://elasticsearch-es-http:9200/customer/_freeze&pretty"

   • Add the following additional action:

     curl -X PUT -u "elastic:<password>" -k "https://elasticsearch-es-http:9200/all/_flush&pretty"

   • Add the following additional action:

     curl -X PUT -u "elastic:<password>" -k "https://elasticsearch-es-http:9200/_snapshot/es_backups/%3Csnapshot-$(uuidgen)-%7Bnow%2Fd%7D%3E?wait_for_completion=true&pretty"
   

3. Click Add.

Create a post-exec backup rule for Elasticsearch

Performing the _freeze and _flush operations on your index before calling the snapshot API ensures flexible and accurate restores by making sure the database isn't currently being written into.

Since you performed a _freezeoperation when you created a backup, you must create a post exec rule to perform _unfreeze operation. The steps below create a post-exec rule which runs _unfreeze on the customer index:

1. From the home page, navigate to Settings >Rules > Add New.

2. In the Add Rule window, provide the following details:

   • Rule name: add a name for your backup rule

   • Pod Selector: add the following app label appname=elasticsearch-app

   • Container: ignore this field

   • Action: add the following action

     curl -X POST -u "elastic:<password>" -k "https://elasticsearch-es-http:9200/customer/_unfreeze&pretty"
   

3. Click Add.

note

If you deployed Elastic Stack using the elastic operator, ensure that you installed the CRDs and the required operator for elastic stack on the destination cluster as well. Also, when you backup and restore, ensure that you backup elastic operator along with the elastic namespace.

Associate the pre-exec and post-exec rules with the Elasticsearch backup

1. Create a backup for your Elasticsearch application.

2. Select the rules you created in the steps above from the pre-exec and post-exec dropdown list:

3. Once you have populated all the fields of Create Backup window, click Create.

note

If your Elasticsearch version is 7.8 and above, pre-exec or post-exec rules are not required to create a backup of your application data. You can simply back up your data without associating any exec rules.

Demo

Watch this short demo of the above information.