Prometheus and Grafana


About Prometheus

Prometheus is an opensource monitoring and alerting toolkit. Prometheus consists of several components some of which are listed below. - The Prometheus server which scrapes(collects) and stores time series data based on a pull mechanism. - A rules engine which allows generation of Alerts based on the scraped metrices. - An alertmanager for handling alerts. - Multiple integrations for graphing and dashboarding.

This document walks you through the steps required to monitor your Portworx cluster with Prometheus and Grafana. Portworx natively supports Prometheus since it exposes the metrics on a REST endpoint. Then, Prometheus can scrape this endpoint.

The following instructions allows you to monitor Portworx via Prometheus and allow the Alertmanager to provide alerts based on configured rules.

The Prometheus Operator creates, configures and manages a prometheus cluster.

The prometheus operator manages 3 customer resource definitions namely: - Prometheus: The Prometheus CRD defines a Prometheus setup to be run on a Kubernetes cluster. The Operator creates a Statefulset for each definition of the Prometheus resource.

  • ServiceMonitor: The ServiceMonitor CRD allows the definition of how Kubernetes services could be monitored based on label selectors. The Service abstraction allows Prometheus to in turn monitor underlying Pods.

  • Alertmanager: The Alertmanager CRD allows the definition of an Alertmanager instance within the Kubernetes cluster. The alertmanager expects a valid configuration in the form of a secret called alertmanager-name.

About Grafana

Grafana is a dashboarding and visualization tool with integrations to several timeseries datasources. It is used to create dashboards for the monitoring data with customizable visualizations. We would use Prometheus as the source of data to view Portworx monitoring metrics.

Prerequisites

  • A running Portworx cluster.

Installation

Install the Prometheus Operator

Download prometheus-operator.yaml and apply it:

kubectl apply -f <prometheus-operator.yaml>

Install the Service Monitor

Create a file named service-monitor.yaml with the below contents:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  namespace: kube-system
  name: portworx-prometheus-sm
  labels:
    name: portworx-prometheus-sm
spec:
  selector:
    matchLabels:
      name: portworx
  namespaceSelector:
    any: true
  endpoints:
  - port: px-api
    targetPort: 9001

Next, apply the spec:

kubectl apply -f <service-monitor.yaml>

Install the Alertmanager

Create a file named alertmanager.yaml with the following contents and create a secret from it. Make sure you add the relevant email addresses in the below config.

global:
  # The smarthost and SMTP sender used for mail notifications.
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: '<sender-email-address>'
  smtp_auth_username: "<sender-email-address>"
  smtp_auth_password: '<sender-email-password>'
route:
  group_by: [Alertname]
  # Send all notifications to me.
  receiver: email-me
receivers:
- name: email-me
  email_configs:
  - to: <receiver-email-address>
    from: <sender-email-address>
    smarthost: smtp.gmail.com:587
    auth_username: "<sender-email-address>"
    auth_identity: "<sender-email-address>"
    auth_password: "<sender-email-password>"
## Edit the file and create a secret with it using the following command
kubectl create secret generic alertmanager-portworx --from-file=alertmanager.yaml -n kube-system

Create a file named alertmanager-cluster.yaml with the below contents:

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  name: portworx #This name is important since the Alertmanager pods wont start unless a secret named alertmanager-${ALERTMANAGER_NAME} is created. in this case if would expect alertmanager-portworx secret in the kube-system namespace
  namespace: kube-system
  labels:
    alertmanager: portworx
spec:
  replicas: 3

Now, apply the spec on your cluster:

kubectl apply -f alertmanager-cluster.yaml

Create a file named alertmanager-service.yaml with the following contents:

apiVersion: v1
kind: Service
metadata:
  name: alertmanager-portworx
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - name: web
    port: 9093
    protocol: TCP
    targetPort: web
  selector:
    alertmanager: portworx

Apply the spec:

kubectl apply -f alertmanager-service.yaml

Install Prometheus

Download prometheus-rules.yaml and apply it:

kubectl apply -f prometheus-rules.yaml

Download prometheus-cluster.yaml and apply it:

kubectl apply -f prometheus-cluster.yaml

Post Install verification

Prometheus access details

Prometheus can be be accessed using a NodePort service.

First get the node port that prometheus is using

  kubectl get svc -n kube-system prometheus

Navigate to the Prometheus web UI by going to http://<master_ip>:<service_nodeport>. You should be able to navigate to the Targets and Rules section of the Prometheus dashboard which lists the Portworx cluster endpoints as well as the Alerting rules as specified earlier.

Installing Grafana

Download grafana-deployment.yaml and apply it:

kubectl apply -f grafana-deployment.yaml

Grafana access details

Grafana can be be accessed using a NodePort service.

First get the node port that grafana is using

  kubectl get svc -n kube-system grafana

Access the Grafana dashboard by navigating to http://<master_ip>:<service_nodeport>. You would need to create a datasource for the Portworx Grafana dashboard metrics to be populated. Navigate to Configurations –> Datasources. Create a datasource named prometheus. Enter the Prometheus endpoint as obtained in the install verification step for Prometheus from the above section.

grafanadatasource

Post install verification

Select the Portworx volume metrics dashboard on Grafana to view the Portworx metrics. grafanadashboard



Last edited: Thursday, Dec 12, 2019