Configure monitoring for AKS
To ensure your storage infrastructure's health, performance, and reliability, it's crucial to monitor your Portworx cluster. The monitoring approach varies depending on your deployment environment. For the majority of setups, Portworx's integrated Prometheus and Grafana deployment runs by default.
Monitoring your Portworx cluster in an Kubernetes environment involves leveraging technologies like Prometheus, Alertmanager, and Grafana. These tools help in collecting data, managing alerts, and visualizing metrics.
- Prometheus: Collects metrics, essential for Autopilot to identify and respond to conditions like storage capacity issues, enabling automatic PVC expansion and storage pool scaling.
- Alertmanager: Manages alerts, integrating with Autopilot for timely notifications about critical cluster conditions.
- Grafana: Visualizes data from Prometheus in an easy-to-understand format, aiding in quick decision-making for cluster management.
Each tool plays a vital role in ensuring efficient, automated monitoring and response within your cluster.
Verify monitoring using Prometheus
You can monitor your Portworx cluster using Prometheus. Portworx deploys Prometheus by default, but you can verify the deployment:
-
Verify that Prometheus pods are running by entering the following
kubectl get pods
command in the namespace where you deployed Portworx. For example:kubectl -n kube-system get pods -A | grep -i prometheus
kube-system prometheus-px-prometheus-0 2/2 Running 0 59m
kube-system px-prometheus-operator-59b98b5897-9nwfv 1/1 Running 0 60m -
Verify that the Prometheus
px-prometheus
andprometheus operated
services exist by entering the following command:kubectl -n kube-system get service | grep -i prometheus
prometheus-operated ClusterIP None <none> 9090/TCP 63m
px-prometheus ClusterIP 10.99.61.133 <none> 9090/TCP 63m
Set up Alertmanager
Prometheus Alertmanager handles alerts sent from the Prometheus server based on rules you set. If any Prometheus rule is triggered, Alertmanager sends a corresponding notification to the specified receivers. You can configure these receivers using an Alertmanager config file. Perform the following steps to configure and enable Alertmanager:
-
Create a valid Alertmanager configuration file and name it
alertmanager.yaml
. The following is a sample for Alertmanager, and the settings used in your environment may be different:global:
# The smarthost and SMTP sender used for mail notifications.
smtp_smarthost: 'smtp.gmail.com:587'
smtp_from: 'username@company.com'
smtp_auth_username: "username@company.com"
smtp_auth_password: 'xyxsy'
route:
group_by: [Alertname]
# Send all notifications to me.
receiver: email-me
receivers:
- name: email-me
email_configs:
- to: username@company.com
from: username@company.com
smarthost: smtp.gmail.com:587
auth_username: "username@company.com"
auth_identity: "username@company.com"
auth_password: "username@company.com" -
Create a secret called
alertmanager-portworx
in the same namespace as your StorageCluster object:kubectl -n kube-system create secret generic alertmanager-portworx --from-file=alertmanager.yaml=alertmanager.yaml
-
Edit your StorageCluster object to enable Alertmanager:
kubectl -n kube-system edit stc <px-cluster-name>
apiVersion: core.libopenstorage.org/v1
kind: StorageCluster
metadata:
name: portworx
namespace: kube-system
monitoring:
prometheus:
enabled: true
exportMetrics: true
alertManager:
enabled: true -
Verify that the Alertmanager pods are running using the following command:
kubectl -n kube-system get pods | grep -i alertmanager
alertmanager-portworx-0 2/2 Running 0 4m9s
alertmanager-portworx-1 2/2 Running 0 4m9s
alertmanager-portworx-2 2/2 Running 0 4m9snoteTo view the complete list of out-of-the-box default rules, see step 7 below.
Access the Alertmanager UI
To access the Alertmanager UI and view the Alertmanager Status and alerts, you need to set up port forwarding and browse to the specified port. In this example, port forwarding is provided for ease of access to the Alertmanager service from your local machine using the port 9093.
-
Set up port forwarding:
kubectl -n kube-system port-forward service/alertmanager-portworx --address=<masternodeIP> 9093:9093
-
Access Prometheus UI by browsing to
http://<masternodeIP>:9093/#/status
Portworx Central on-premises includes Grafana and Portworx dashboards natively, which you can use to monitor your Portworx cluster. Refer to the Portworx Central documentation for further details.
Access the Prometheus UI
To access the Prometheus UI to view Status, Graph and default Alerts, you also need to set up port forwarding and browse to the specified port. In this example, Port forwarding is provided for ease of access to the Prometheus UI service from your local machine using the port 9090.
-
Set up port forwarding:
kubectl -n kube-system port-forward service/px-prometheus 9090:9090
-
Access the Prometheus UI by browsing to
http://localhost:9090/alerts
.
View provided Prometheus rules
To view the complete list of out-of-the-box default rules used for event notifications, perform the following steps.
-
Get the Prometheus rules:
kubectl -n kube-system get prometheusrules
NAME AGE
portworx 46d -
Save the Prometheus rules to a YAML file:
kubectl -n kube-system get prometheusrules portworx -o yaml > prometheusrules.yaml
-
View the contents of the file:
cat prometheusrules.yaml
Configure Grafana
You can connect to Prometheus using Grafana to visualize your data. Grafana is a multi-platform open source analytics and interactive visualization web application. It provides charts, graphs, and alerts.
-
Enter the following commands to download the Grafana dashboard and datasource configuration files:
wget https://docs.portworx.com/samples/portworx-enterprise/k8s/pxc/grafana-dashboard-config.yaml
wget https://docs.portworx.com/samples/portworx-enterprise/k8s/pxc/grafana-datasource.yaml
-
Create a configmap for the dashboard and data source:
kubectl -n <px-namespace> create configmap grafana-dashboard-config --from-file=grafana-dashboard-config.yaml
kubectl -n <px-namespace> create configmap grafana-source-config --from-file=grafana-datasource.yaml
-
Download and install Grafana dashboards using the following commands:
wget https://docs.portworx.com/samples/portworx-enterprise/k8s/pxc/portworx-cluster-dashboard.json
wget https://docs.portworx.com/samples/portworx-enterprise/k8s/pxc/portworx-node-dashboard.json
wget https://docs.portworx.com/samples/portworx-enterprise/k8s/pxc/portworx-volume-dashboard.json
wget https://docs.portworx.com/samples/portworx-enterprise/k8s/pxc/portworx-performance-dashboard.json
wget https://docs.portworx.com/samples/portworx-enterprise/k8s/pxc/portworx-etcd-dashboard.jsonkubectl -n <px-namespace> create configmap grafana-dashboards \
--from-file=portworx-cluster-dashboard.json \
--from-file=portworx-performance-dashboard.json \
--from-file=portworx-node-dashboard.json \
--from-file=portworx-volume-dashboard.json \
--from-file=portworx-etcd-dashboard.json -
Enter the following command to download and install the Grafana YAML file:
kubectl apply -f https://docs.portworx.com/samples/portworx-enterprise/k8s/pxc/grafana.yaml
-
Verify if the Grafana pod is running using the following command:
kubectl -n <px-namespace> get pods | grep -i grafana
grafana-7d789d5cf9-bklf2 1/1 Running 0 3m12s
-
Access Grafana by setting up port forwarding and browsing to the specified port. In this example, port forwarding is provided for ease of access to the Grafana service from your local machine using the port 3000:
kubectl -n <px-namespace> port-forward service/grafana 3000:3000
-
Navigate to Grafana by browsing to
http://localhost:3000
. -
Enter the default credentials to log in.
- login:
admin
- password:
admin
- login: