Version: 2.10

Configure Grafana

This topic describes how to configure Grafana to monitor your Portworx Backup clusters. Grafana enables you to visualize metrics, create custom dashboards, set up alerting, and aggregate data from multiple Portworx Backup instances for a multi-tenant view.

Prerequisites

Portworx Backup 2.10.1 or later. For earlier versions (2.7.x or earlier), refer to the Monitor PXB Clusters.
Prometheus configured and scraping Portworx Backup metrics. You can use either:
- Portworx Backup Prometheus
- Your own Prometheus stack
kubectl access to your Portworx Backup cluster

Use Cases

Grafana provides several powerful capabilities for monitoring Portworx Backup:

Use Case	Description
Custom Dashboards	Visualize backup, restore, and cluster metrics with pre-built or custom dashboards
Custom Querying	Use PromQL to query specific metrics and create custom visualizations
Alerting	Configure Grafana alerts for backup failures, cluster disconnections, and other critical events
Multi-Tenant View	Scrape and aggregate metrics from multiple Portworx Backup instances for centralized monitoring and alerting

Install and Configure Grafana

Create Storage Resources

Create a storage class for Grafana and persistent volumes with the grafana-data, grafana-dashboard, grafana-source-config, and grafana-extensions names:

kubectl apply -f - <<'_EOF'
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
    name: px-grafana-sc
provisioner: kubernetes.io/portworx-volume
parameters:
  repl: "3"
  priority_io: "high"
allowVolumeExpansion: true
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: grafana-data
  namespace: <pxb-namespace>
  annotations:
    volume.beta.kubernetes.io/storage-class: px-grafana-sc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: grafana-dashboard
  namespace: <pxb-namespace>
  annotations:
    volume.beta.kubernetes.io/storage-class: px-grafana-sc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: grafana-source-config
  namespace: <pxb-namespace>
  annotations:
    volume.beta.kubernetes.io/storage-class: px-grafana-sc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: grafana-extensions
  namespace: <pxb-namespace>
  annotations:
    volume.beta.kubernetes.io/storage-class: px-grafana-sc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

note

In this storage class:

The provisioner parameter is set to kubernetes.io/portworx-volume. For details about the Portworx-specific parameters, refer to the Portworx Volume section of the Kubernetes website.
Three replicas of each volume will be created.

Deploy Grafana

Enter the following command to install Grafana:

note

If your cluster is on a cloud provider, then follow the instructions in Step 3 to export it to a node port or load balancer.

kubectl apply -n <pxb-namespace> -f - <<'_EOF'
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: <pxb-namespace>
  labels:
    app: grafana
spec:
  type: ClusterIP
  ports:
    - port: 3000
  selector:
    app: grafana
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: <pxb-namespace>
  labels:
    app: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
      fsGroup: 2000
      containers:
        - image: docker.io/portworx/grafana:7.5.16
          name: grafana
          imagePullPolicy: Always
          resources:
            limits:
              cpu: 100m
              memory: 100Mi
            requests:
              cpu: 100m
              memory: 100Mi
          readinessProbe:
            httpGet:
              path: /login
              port: 3000
          volumeMounts:
            - name: grafana
              mountPath: /etc/grafana/provisioning/dashboard
              readOnly: false
            - name: grafana-dash
              mountPath: /var/lib/grafana/dashboards
              readOnly: false
            - name: grafana-source-cfg
              mountPath: /etc/grafana/provisioning/datasources
              readOnly: false
            - name: grafana-plugins
              mountPath: /var/lib/grafana/plugins
              readOnly: false
      volumes:
      - name: grafana
        persistentVolumeClaim:
          claimName: grafana-data
      - name: grafana-dash
        persistentVolumeClaim:
          claimName: grafana-dashboard
      - name: grafana-source-cfg
        persistentVolumeClaim:
          claimName: grafana-source-config
      - name: grafana-plugins
        persistentVolumeClaim:
          claimName: grafana-extensions

note

In this deployment, the volumes section references the PVCs you created in the previous step.

Access Grafana

Enter the following kubectl port-forward command to forward all connections made to localhost:3000 to svc/grafana:3000:
```
kubectl port-forward svc/grafana --namespace <pxb-namespace> --address 0.0.0.0 3000
```
Alternatively, if your cluster is on a cloud provider, perform one of the following to export to the nodeport or loadbalancer:
- To export to the nodeport, navigate to Deploy Grafana, change the following parameter in Step 2:
```
type: NodePort
```
  note
  The cluster node must be accessible using an external IP.
- To export to the loadbalancer, create an ingress rule:
  
  note
  The following example uses the Portworx Backup Prometheus service (px-backup-prometheus). If you are using your own Prometheus stack, replace the serviceName with your Prometheus service name.
```
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: grafana-ingress
  namespace: <pxb-namespace>
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: px-backup-prometheus 
          servicePort: 9090
```

Configure Prometheus Data Source

Configure a Prometheus data source in Grafana. The configuration depends on your Prometheus setup:
- If you are using Portworx Backup Prometheus: You must first expose the Prometheus metrics endpoint. Follow the instructions in Expose Portworx Backup Prometheus to create a NodePort service and retrieve the authentication credentials.
- If you are using your own Prometheus stack: Ensure that your Prometheus instance is accessible from Grafana. For details on exposing the metrics endpoint, see Configure your Own Prometheus.
After configuring the endpoint, follow the instructions in the Prometheus documentation to create a Prometheus data source named px-backup. Replace the HTTP URL http://localhost:9090/ with the appropriate URL for your setup:
- For Portworx Backup Prometheus, use the exposed metrics endpoint constructed using the master node IP and NodePort (e.g., http://<master-ip>:<nodeport>).
- For your own Prometheus use the exposed metrics endpoint where your Prometheus instance is accessible.

Import Dashboards

Follow the instructions in the Grafana documentation to import the Portworx Backup dashboard JSON into your Grafana instance.
- Click here to download a sample Grafana dashboard JSON for the Portworx Backup Overview, which you can import into your Grafana instance.
- Click here to download a sample Grafana dashboard JSON containing Backup Info, ready for import into your Grafana instance.

Custom Querying in Grafana

Grafana allows you to create custom queries using PromQL to visualize specific Portworx Backup metrics. You can:

Build custom panels with specific metrics from the Portworx Backup metrics endpoint
Create calculated fields and aggregations
Filter data by labels such as cluster name, backup name, or namespace

For a complete list of available metrics, see the Portworx Backup Metrics Reference.

Example Queries

Query	Description
`px_backup_total_backups`	Total number of backups
`px_backup_failed_backups`	Number of failed backups
`rate(px_backup_total_backups[1h])`	Backup rate over the last hour

Alerting in Grafana

Grafana can alert you when backup metrics cross defined thresholds. To configure alerting:

Navigate to Alerting > Alert rules in Grafana
Create a new alert rule based on Portworx Backup metrics
Define conditions (e.g., alert when px_backup_failed_backups > 0)
Configure notification channels (email, Slack, PagerDuty, etc.)

Multi-Tenant Monitoring

For organizations running multiple Portworx Backup instances across different clusters or environments, Grafana can aggregate metrics for a unified view.

Architecture Overview

Configure Multiple Data Sources

In Grafana, navigate to Configuration > Data Sources
Add each Prometheus instance as a separate data source. The URL and authentication settings depend on your Prometheus setup:
- For Portworx Backup Prometheus instances: Follow the instructions in Expose Portworx Backup Prometheus to expose each instance and retrieve the authentication credentials.
- For your own Prometheus instances: Use the URL where each Prometheus instance is accessible. Authentication depends on your Prometheus configuration.
Example data source configuration:
- Name: pxb-cluster-a, URL: http://<prometheus-1-endpoint>:<port>
- Name: pxb-cluster-b, URL: http://<prometheus-2-endpoint>:<port>
- Name: pxb-cluster-c, URL: http://<prometheus-3-endpoint>:<port>
Use dashboard variables to switch between clusters or create mixed queries

Create Aggregated Dashboards

Use Grafana's Mixed data source to query multiple Prometheus instances in a single panel, enabling:

Cross-cluster backup status overview
Aggregated success/failure rates across all environments
Unified alerting for all Portworx Backup instances

Prerequisites​

Use Cases​

Install and Configure Grafana​

Create Storage Resources​

Deploy Grafana​

Access Grafana​

Configure Prometheus Data Source​

Import Dashboards​

Custom Querying in Grafana​

Example Queries​

Alerting in Grafana​

Multi-Tenant Monitoring​

Architecture Overview​

Configure Multiple Data Sources​

Create Aggregated Dashboards​

Prerequisites

Use Cases

Install and Configure Grafana

Create Storage Resources

Deploy Grafana

Access Grafana

Configure Prometheus Data Source

Import Dashboards

Custom Querying in Grafana

Example Queries

Alerting in Grafana

Multi-Tenant Monitoring

Architecture Overview

Configure Multiple Data Sources

Create Aggregated Dashboards