For Portworx 2.7.x or earlier: Monitor PXB Clusters

This topic describes how you can monitor your Portworx Backup clusters with Prometheus and Grafana. This section is exclusively for the users who want to use external Prometheus and Grafana for extensive monitoring data of PXB clusters.

note

If you are referring to this topic, opt out of using PXB Prometheus stack for monitoring.

Configure Service Monitor

If Grafana already exists in the cluster where PXB is deployed, then you just have to configure service monitor to monitor PXB clusters. Apply the following spec in <pxb-namespace> for extensive monitoring:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    name: px-backup-sm
  name: px-backup-sm
  namespace: <pxb-namespace>
spec:
  endpoints:
  - metricRelabelings:
    - action: labeldrop
      regex: (instance|pod)
    - action: drop
      regex: process_.*
      sourceLabels:
      - __name__
    - action: drop
      regex: go_.*
      sourceLabels:
      - __name__
    - action: drop
      regex: grpc_.*
      sourceLabels:
      - __name__
    port: rest-api
    targetPort: 10001
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      app: px-backup

Prerequisites

• A Portworx Backup cluster
• Environment required to install Prometheus and Grafana
• You must have kubectl access to your Portworx Backup cluster

Below topic explains the deployment of monitoring stack in Portworx Backup namespace. If Portworx Backup is deployed in a different namespace, please ensure to modify the namespace wherever required.

Install and configure Prometheus

info

For Portworx Backup 2.7.0 or above, see Configure your Own Prometheus which covers OpenShift Clusters, Rancher, and Kubernetes Vanilla environments.

1. (Optional) Enter the following combined spec and kubectl command to install the Prometheus Operator:

   Skip this step if you have not configured your own Prometheus stack in Portworx Backup version 2.7.0 and above. If no, execute the following steps.

   kubectl apply -f - <<'_EOF'
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: prometheus-operator
     namespace: <pxb-namespace>
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: prometheus-operator
   subjects:
     - kind: ServiceAccount
       name: prometheus-operator
       namespace: <pxb-namespace>
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: prometheus-operator
     namespace: <pxb-namespace>
   rules:
     - apiGroups:
         - extensions
       resources:
         - thirdpartyresources
       verbs: ["*"]
     - apiGroups:
         - apiextensions.k8s.io
       resources:
         - customresourcedefinitions
       verbs: ["*"]
     - apiGroups:
         - monitoring.coreos.com
       resources:
         - alertmanagers
         - prometheuses
         - prometheuses/finalizers
         - servicemonitors
         - prometheusrules
         - podmonitors
         - thanosrulers
         - alertmanagerconfigs
         - probes
       verbs: ["*"]
     - apiGroups:
         - apps
       resources:
         - statefulsets
       verbs: ["*"]
     - apiGroups: [""]
       resources:
         - configmaps
         - secrets
       verbs: ["*"]
     - apiGroups: [""]
       resources:
         - pods
       verbs: ["list", "delete"]
     - apiGroups: [""]
       resources:
         - services
         - endpoints
       verbs: ["get", "create", "update", "delete"]
     - apiGroups: [""]
       resources:
         - nodes
       verbs: ["list", "watch"]
     - apiGroups: [""]
       resources:
         - namespaces
       verbs: ["list", "watch", "get"]
   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: prometheus-operator
     namespace: <pxb-namespace>
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       k8s-app: prometheus-operator
     name: prometheus-operator
     namespace: <pxb-namespace>
   spec:
     selector:
       matchLabels:
         k8s-app: prometheus-operator
     replicas: 1
     template:
       metadata:
         labels:
           k8s-app: prometheus-operator
       spec:
         containers:
           - args:
               - --kubelet-service=kube-system/kubelet
               - --prometheus-config-reloader=docker.io/portworx/prometheus-config-reloader:v0.56.3
               - --namespaces=<pxb-namespace>
             name: prometheus-operator
             image: docker.io/portworx/prometheus-operator:v0.56.3
             ports:
               - containerPort: 8080
                 name: http
             resources:
               limits:
                 cpu: 200m
                 memory: 100Mi
               requests:
                 cpu: 100m
                 memory: 50Mi
         securityContext:
           runAsNonRoot: true
           runAsUser: 65534
         serviceAccountName: prometheus-operator

2. To grant Prometheus access to the metrics API, create the ClusterRole, ClusterRoleBinding, Service, and ServiceAccount Kubernetes objects:

   kubectl apply -f - <<'_EOF'
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: px-backup-prometheus
     namespace: <pxb-namespace>
   rules:
     - apiGroups:
       - ""
       resources:
       - nodes
       - services
       - endpoints
       - pods
       verbs:
       - get
       - list
       - watch
     - apiGroups:
       - ""
       resources:
       - configmaps
       verbs:
       - get
     - nonResourceURLs:
       - /metrics
       - /federate
       verbs:
       - get
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: px-backup-prometheus
     namespace: <pxb-namespace>
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: px-backup-prometheus
   subjects:
     - kind: ServiceAccount
       name: px-backup-prometheus
       namespace: <pxb-namespace>
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: px-backup-prometheus
     namespace: <pxb-namespace>
   spec:
     type: ClusterIP
     ports:
       - name: web
         port: 9090
         protocol: TCP
         targetPort: 9090
     selector:
       prometheus: px-backup-prometheus
   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: px-backup-prometheus
     namespace: <pxb-namespace>

3. To specify the monitoring rules for Portworx Backup, create a ServiceMonitor object by entering the following combined spec and kubectl command:

   kubectl apply -f - <<'_EOF'
   ---
   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     namespace: <pxb-namespace>
     name: px-backup-prometheus-sm
     labels:
       name: px-backup-prometheus-sm
   spec:
     selector:
       matchLabels:
         app: px-backup
     namespaceSelector:
       any: true
     endpoints:
       - port: rest-api
         targetPort: 10001

4. Apply Prometheus specs for Portworx Backup metrics:

   kubectl apply -f - <<'_EOF'
   ---
   apiVersion: monitoring.coreos.com/v1
   kind: Prometheus
   metadata:
     name: px-backup-prometheus
     namespace: <pxb-namespace>
   spec:
     replicas: 2
     logLevel: debug
     serviceAccountName: px-backup-prometheus
     serviceMonitorSelector:
       matchLabels:
         name: px-backup-prometheus-sm