Skip to main content
Version: 2.7

Configure Prometheus and Grafana

This document shows how you can monitor your Portworx Backup cluster with Prometheus and Grafana.

Prerequisites

  • A Portworx Backup cluster
  • You must have kubectl access to your Portworx Backup cluster

Below topic explains the deployment of monitoring stack in Portworx Backup namespace. If Portworx Backup is deployed in a different namespace, please ensure to modify the namespace wherever required.

Install and configure Prometheus

  1. (Optional) Enter the following combined spec and kubectl command to install the Prometheus Operator:

Skip this step if you have not configured your own Prometheus stack in Portworx Backup version 2.7.0

kubectl apply -f - <<'_EOF'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus-operator
namespace: px-backup
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus-operator
subjects:
- kind: ServiceAccount
name: prometheus-operator
namespace: px-backup
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus-operator
namespace: px-backup
rules:
- apiGroups:
- extensions
resources:
- thirdpartyresources
verbs: ["*"]
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs: ["*"]
- apiGroups:
- monitoring.coreos.com
resources:
- alertmanagers
- prometheuses
- prometheuses/finalizers
- servicemonitors
- prometheusrules
- podmonitors
- thanosrulers
- alertmanagerconfigs
- probes
verbs: ["*"]
- apiGroups:
- apps
resources:
- statefulsets
verbs: ["*"]
- apiGroups: [""]
resources:
- configmaps
- secrets
verbs: ["*"]
- apiGroups: [""]
resources:
- pods
verbs: ["list", "delete"]
- apiGroups: [""]
resources:
- services
- endpoints
verbs: ["get", "create", "update"]
- apiGroups: [""]
resources:
- nodes
verbs: ["list", "watch"]
- apiGroups: [""]
resources:
- namespaces
verbs: ["list", "watch", "get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus-operator
namespace: px-backup
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: prometheus-operator
name: prometheus-operator
namespace: px-backup
spec:
selector:
matchLabels:
k8s-app: prometheus-operator
replicas: 1
template:
metadata:
labels:
k8s-app: prometheus-operator
spec:
containers:
- args:
- --kubelet-service=kube-system/kubelet
- --prometheus-config-reloader=docker.io/portworx/prometheus-config-reloader:v0.56.3
- --namespaces=px-backup
name: prometheus-operator
image: docker.io/portworx/prometheus-operator:v0.56.3
ports:
- containerPort: 8080
name: http
resources:
limits:
cpu: 200m
memory: 100Mi
requests:
cpu: 100m
memory: 50Mi
securityContext:
runAsNonRoot: true
runAsUser: 65534
serviceAccountName: prometheus-operator
  1. To grant Prometheus access to the metrics API, create the ClusterRole, ClusterRoleBinding, Service, and ServiceAccount Kubernetes objects:

    kubectl apply -f - <<'_EOF'
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
    name: px-backup-prometheus
    rules:
    - apiGroups:
    - ""
    resources:
    - nodes
    - services
    - endpoints
    - pods
    verbs:
    - get
    - list
    - watch
    - apiGroups:
    - ""
    resources:
    - configmaps
    verbs:
    - get
    - nonResourceURLs:
    - /metrics
    - /federate
    verbs:
    - get
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
    name: px-backup-prometheus
    roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: px-backup-prometheus
    subjects:
    - kind: ServiceAccount
    name: px-backup-prometheus
    namespace: px-backup
    ---
    apiVersion: v1
    kind: Service
    metadata:
    name: px-backup-prometheus
    namespace: px-backup
    spec:
    type: ClusterIP
    ports:
    - name: web
    port: 9090
    protocol: TCP
    targetPort: 9090
    selector:
    prometheus: px-backup-prometheus
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
    name: px-backup-prometheus
    namespace: px-backup
  2. To specify the monitoring rules for Portworx Backup, create a ServiceMonitor object by entering the following combined spec and kubectl command:

    kubectl apply -f - <<'_EOF'
    ---
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
    namespace: px-backup
    name: px-backup-prometheus-sm
    labels:
    name: px-backup-prometheus-sm
    spec:
    selector:
    matchLabels:
    app: px-backup
    namespaceSelector:
    any: true
    endpoints:
    - port: rest-api
    targetPort: 10001
  3. Apply Prometheus specs for Portworx Backup metrics:

    kubectl apply -f - <<'_EOF'
    ---
    apiVersion: monitoring.coreos.com/v1
    kind: Prometheus
    metadata:
    name: px-backup-prometheus
    namespace: px-backup
    spec:
    replicas: 2
    logLevel: debug
    serviceAccountName: px-backup-prometheus
    serviceMonitorSelector:
    matchLabels:
    name: px-backup-prometheus-sm

Install and configure Grafana

  1. Create a storage class for Grafana and persistent volumes with the grafana-data, grafana-dashboard, grafana-source-config, and grafana-extensions names:

    kubectl apply -f - <<'_EOF'
    ---
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
    name: px-grafana-sc
    provisioner: kubernetes.io/portworx-volume
    parameters:
    repl: "3"
    priority_io: "high"
    allowVolumeExpansion: true
    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
    name: grafana-data
    namespace: px-backup
    annotations:
    volume.beta.kubernetes.io/storage-class: px-grafana-sc
    spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
    storage: 1Gi
    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
    name: grafana-dashboard
    namespace: px-backup
    annotations:
    volume.beta.kubernetes.io/storage-class: px-grafana-sc
    spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
    storage: 1Gi
    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
    name: grafana-source-config
    namespace: px-backup
    annotations:
    volume.beta.kubernetes.io/storage-class: px-grafana-sc
    spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
    storage: 1Gi
    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
    name: grafana-extensions
    namespace: px-backup
    annotations:
    volume.beta.kubernetes.io/storage-class: px-grafana-sc
    spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
    storage: 1Gi
    note

    In this storage class:

    • The provisioner parameter is set to kubernetes.io/portworx-volume. For details about the Portworx-specific parameters, refer to the Portworx Volume section of the Kubernetes website.
    • Three replicas of each volume will be created.
  2. Enter the following command to install Grafana:

    note

    If your cluster is on a cloud provider, then follow the instructions in Step 3 to export it to a nodeport or loadbalancer.

    kubectl apply -n px-backup -f - <<'_EOF'
    ---
    apiVersion: v1
    kind: Service
    metadata:
    name: grafana
    labels:
    app: grafana
    spec:
    type: ClusterIP
    ports:
    - port: 3000
    selector:
    app: grafana
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: grafana
    labels:
    app: grafana
    spec:
    replicas: 1
    selector:
    matchLabels:
    app: grafana
    template:
    metadata:
    labels:
    app: grafana
    spec:
    securityContext:
    fsGroup: 2000
    containers:
    - image: docker.io/portworx/grafana:7.5.16
    name: grafana
    imagePullPolicy: Always
    resources:
    limits:
    cpu: 100m
    memory: 100Mi
    requests:
    cpu: 100m
    memory: 100Mi
    readinessProbe:
    httpGet:
    path: /login
    port: 3000
    volumeMounts:
    - name: grafana
    mountPath: /etc/grafana/provisioning/dashboard
    readOnly: false
    - name: grafana-dash
    mountPath: /var/lib/grafana/dashboards
    readOnly: false
    - name: grafana-source-cfg
    mountPath: /etc/grafana/provisioning/datasources
    readOnly: false
    - name: grafana-plugins
    mountPath: /var/lib/grafana/plugins
    readOnly: false
    volumes:
    - name: grafana
    persistentVolumeClaim:
    claimName: grafana-data
    - name: grafana-dash
    persistentVolumeClaim:
    claimName: grafana-dashboard
    - name: grafana-source-cfg
    persistentVolumeClaim:
    claimName: grafana-source-config
    - name: grafana-plugins
    persistentVolumeClaim:
    claimName: grafana-extensions
    note

    In this deployment, the volumes section references the PVCs you created in the previous step.

  3. Enter the following kubectl port-forward command to forward all connections made to localhost:3000 to svc/grafana:3000:

    kubectl port-forward svc/grafana --namespace px-backup --address 0.0.0.0 3000

    Alternatively, if your cluster is on a cloud provider, perform one of the following to export to the nodeport or loadbalancer:

    • To export to the nodeport, navigate to Install and configure Grafana, change the following parameter in Step 2:

      type: NodePort
      note

      The cluster node must be accessible using an external IP.

    • To export to the loadbalancer, create an ingress role:

      apiVersion: extensions/v1beta1
      kind: Ingress
      metadata:
      name: guestbook2
      namespace: px-backup
      annotations:
      kubernetes.io/ingress.class: azure/application-gateway
      spec:
      rules:
      - http:
      paths:
      - backend:
      serviceName: px-backup-prometheus
      servicePort: 9090
  4. Follow the instructions in Prometheus documentation to create a Prometheus data source named px-backup. Please ensure to replace the HTTP URL from http://localhost:9090/ to http://prometheus-operated:9090/.

  1. Follow the instructions in Grafana documentation to import the Portworx Backup dashboard JSON file.
Was this page helpful?