This repository is an example of how to make use of kube-prometheus-stack and loki-stack to monitor Flux.
Components:
- kube-state-metrics - generates metrics about the state of the Flux objects
- Prometheus Operator - manages Prometheus clusters atop Kubernetes
- Prometheus - collects and stores metrics from the Flux controllers and kube-state-metrics
- Promtail - collects the logs from the Flux controllers
- Loki - stores the logs collected by Promtail
- Grafana dashboards - displays the Flux control plane resource usage, reconciliation stats and logs
For a quick local test, you can use Kubernetes kind. Any other Kubernetes setup will work as well though.
Create a cluster called test
with the kind CLI:
kind create cluster --name test
In order to follow this guide you'll need a GitHub account and a
personal access token
that can create repositories (check all permissions under repo
).
Add the GitHub PAT and username to your shell environment:
export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>
Fork this repository on your personal account and clone it locally:
git clone https://github.com/${GITHUB_USER}/flux2-monitoring-example.git
cd flux2-monitoring-example
Install the Flux controllers on the test cluster:
flux bootstrap github \
--owner=${GITHUB_USER} \
--repository=flux2-monitoring-example \
--branch=main \
--personal \
--path=clusters/test
Wait for Flux to deploy the monitoring stack with:
flux get kustomizations --watch
After Flux has finished reconciling, you can list the pods in the monitoring namespace with:
$ kubectl -n monitoring get po
NAME READY
kube-prometheus-stack-grafana-5c976ff4cf-xgmwm 3/3
kube-prometheus-stack-kube-state-metrics-5dcf4c4697-jvlvh 1/1
kube-prometheus-stack-operator-75f9fdcbf6-98zmh 1/1
kube-prometheus-stack-prometheus-node-exporter-j4vhb 1/1
loki-stack-0 1/1
loki-stack-promtail-dcg64 1/1
prometheus-kube-prometheus-stack-prometheus-0 2/2
To access Grafana, start port forward in a separate shell:
kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80
Navigate to http://localhost:3000
in your browser and login with user admin
and password flux
.
Flux dashboards: